MARINA-P improves non-smooth federated optimization with adaptive stepsizes.
problem Non-smooth federated optimization in machine learning applications.
method Extends EF21-P and MARINA-P to non-smooth convex setting, proving optimal convergence rate and communication complexity bounds.
result MARINA-P achieves O ( 1 / T ) O(1/\sqrt{T}) O ( 1/ T ) convergence rate and communication complexity matching classical subgradient methods. New algorithm finds local minima in non-convex, non-smooth problems.
problem Finding local minimizers in non-convex and non-smooth optimization.
method Perturbed Proximal Descent, tailored for non-smooth cases.
result First known results for non-smooth optimization.
New algorithms optimize non-smooth, non-convex objectives with improved complexity.
problem Optimizing non-smooth, non-convex stochastic objectives.
method Reduction to online learning, applying optimistic online learning techniques.
result Improved complexity for finding ( δ , ε ) (δ,ε) ( δ , ε ) -stationary points. Paper tackles private optimization for non-smooth objectives efficiently.
problem Private stochastic convex optimization for non-smooth objectives.
method Noisy mirror descent algorithm.
result Achieves optimal rates in statistical complexity and number of queries.
This work speeds up hyperparameter selection for non-smooth convex models using implicit differentiation.
problem Optimizing hyperparameters of non-smooth convex models.
method Implicit differentiation of proximal gradient and coordinate descent methods.
result Implicit differentiation can speed up hyperparameter optimization, especially for non-smooth problems.
New SPS variant improves non-smooth optimization without small gradients.
problem Improving non-smooth optimization without small gradients.
method Safeguarded Stochastic Polyak Step Size (SPS s a f e _{safe} s a f e ) for non-smooth optimization. result Rigorous convergence guarantees for non-smooth convex optimization without strong assumptions.
Optimizes deep learning pipelines with novel algorithms for smooth and non-smooth functions.
problem Optimizing deep learning pipelines for smooth and non-smooth functions.
method Provided matching lower and upper bounds for smooth convex and non-convex functions, and developed PPRS for non-smooth convex functions.
result PPRS achieves near-linear speed-up and convergence time for non-smooth non-convex problems.
The paper explores various stationarity concepts in non-smooth optimization.
problem Understanding stationarity in non-smooth optimization problems.
method Introduction and discussion of different stationarity concepts for non-convex non-smooth functions.
result Clarification of the relationship among different stationarity concepts and their relevance in iterative methods.
Optimal private ERM and SCO with subquadratic gradient complexity.
problem Private optimization of non-smooth convex functions.
method Subquadratic gradient complexity algorithm using subsampling and smoothing.
result Achieved optimal excess empirical risk and population loss.
Expanding FCCO to non-smooth weakly-convex problems, improving deep learning performance.
problem Addressing the limitations of current FCCO methods by tackling non-smooth weakly-convex problems.
method Developed a single-loop algorithm for non-smooth weakly-convex FCCO and extended it to tri-level problems.
result Established the complexity for finding ε-stationary points in the Moreau envelop of the objective function.
Improved non-smooth optimization methods achieve faster convergence rates.
problem Non-smooth optimization problems, especially in ℓ ∞ \ell_\infty ℓ ∞ and ℓ 1 \ell_1 ℓ 1 -SVM. method Higher-order accelerated methods, leveraging recent advances in smooth convex optimization.
result Achieved O ( ε − 4 / 5 ) O(ε^{-4/5}) O ( ε − 4/5 ) iteration complexity for ℓ ∞ \ell_\infty ℓ ∞ regression, breaking previous barriers. Inertial methods solve non-convex non-smooth optimization problems efficiently.
problem Non-convex non-smooth optimization problems.
method Inertial block proximal methods for solving these problems.
result The methods converge globally under certain conditions and perform well in applications like NMF.
A new method solves convex optimization on curved spaces.
problem Optimization on curved spaces with non-smooth functions.
method Convex bundle method on Riemannian manifolds.
result The method converges to a minimizer under mild conditions.
A new optimization method, BPM, converges linearly in non-convex, non-smooth problems.
problem Non-smooth and non-convex optimization challenges.
method Ball-Proximal Point Method (BPM), inspired by Proximal Point Method (PPM).
result BPM converges linearly and in a finite number of steps in non-convex, non-smooth problems.
New sampling algorithm for non-smooth potentials.
problem Sampling from non-smooth potentials.
method Proximal algorithm based on rejection sampling.
result Achieves better complexity than existing methods.
New methods improve convergence in non-convex non-smooth learning problems.
problem Sparse learning from high-dimensional data with non-convex, non-smooth regularizers.
method Stochastic proximal gradient methods with arbitrary sampling.
result Independent sampling improves performance over uniform sampling.
Advances smooth over-parameterization for solving non-smooth optimization problems.
problem Non-smooth optimization with structural constraints in imaging and machine learning.
method Smooth over-parameterization of non-smooth problems, using gradient descent and mirror descent.
result Gradient descent on the reformulated smooth problem converges efficiently without parameter tuning.
New algorithms for differentially private optimization in convex and non-convex settings with near-optimal rates.
problem Differentially private optimization in convex and non-convex settings.
method Developed algorithms for convex and non-convex settings with near-optimal excess population risk.
result Achieved near-optimal rates in near-linear time for convex settings and nearly dimension independent rates for non-convex settings.
Online SGD from random init solves non-smooth, non-convex phase retrieval.
problem Solving phase retrieval with non-smooth, non-convex loss functions.
method Online stochastic gradient descent (SGD) with constant step size, starting from arbitrary initialization.
result SGD converges from arbitrary initializations for the amplitude squared loss objective.
Stochastic Gradient Descent (SGD) is one of the simplest and most popular stochastic optimization methods. While it has already been theoretically studied for decades, the classical analysis usually required non-trivial smoothness assumptions, which do not apply to many modern applications of SGD with non-smooth object…
New algorithm samples efficiently from complex composite potentials.
problem Sampling from densities with smooth and non-smooth components.
method Metropolis-Hastings framework with proximal-based proposal.
result Mixes to target density in O ( d log ( d / ε ) ) O(d \log (d/\varepsilon)) O ( d log ( d / ε )) iterations. This paper focuses on convex constrained optimization problems, where the solution is subject to a convex inequality constraint. In particular, we aim at challenging problems for which both projection into the constrained domain and a linear optimization under the inequality constraint are time-consuming, which render …
In this paper, we develop a novel {\bf ho}moto{\bf p}y {\bf s}moothing (HOPS) algorithm for solving a family of non-smooth problems that is composed of a non-smooth term with an explicit max-structure and a smooth term or a simple non-smooth term whose proximal mapping is easy to compute. The best known iteration compl…
We consider the problem of finding critical points of functions that are non-convex and non-smooth. Studying a fairly broad class of such problems, we analyze the behavior of three gradient-based methods (gradient descent, proximal update, and Frank-Wolfe update). For each of these methods, we establish rates of conver…
Paper relaxes SGD privacy and generalization guarantees for non-smooth convex losses.
problem Privacy and generalization in SGD for non-smooth convex losses.
method Relaxes Lipschitz and strong smoothness assumptions to Hölder smoothness, proving ( ε , δ ) (ε,δ) ( ε , δ ) -DP and optimal excess risk. result Noisy SGD with α α α -Hölder smooth losses achieves optimal excess risk with linear gradient complexity for α ≥ 1 / 2 α \geq 1/2 α ≥ 1/2 . Paper proposes ZO-SMD for MERO, achieving optimal convergence rates.
problem Minimizing excess risk across all test distributions.
method Zeroth-order stochastic mirror descent algorithm for both smooth and non-smooth MERO.
result Converges at optimal rates of O ( 1 / t ) \mathcal{O}(1/\sqrt{t}) O ( 1/ t ) for estimates and optimization errors. New algorithm for robust high-dimensional linear regression is both fast and statistically optimal.
problem Challenges in high-dimensional linear regression under heavy-tailed noise or outliers.
method Projected sub-gradient descent algorithm for sparse and low-rank regression problems.
result Algorithm achieves linear convergence and statistical optimality under various noise conditions.
Given a convex optimization problem and its dual, there are many possible first-order algorithms. In this paper, we show the equivalence between mirror descent algorithms and algorithms generalizing the conditional gradient method. This is done through convex duality, and implies notably that for certain problems, such…
Improved stochastic gradient descent analysis for non-smooth convex functions.
problem Minimizing non-smooth, non-differentiable convex functions.
method Stochastic gradient descent with suffix averaging method analysis.
result Error rate of final iterate is O ( log ( T ) / T ) O(\log(T)/T) O ( log ( T ) / T ) with high probability. We analyze convergence rates of stochastic optimization procedures for non-smooth convex optimization problems. By combining randomized smoothing techniques with accelerated gradient methods, we obtain convergence rates of stochastic optimization procedures, both in expectation and with high probability, that have opti…
New bounds explain deterministic non-smooth deep nets without large Lipschitz constants.
problem Challenges in explaining generalization of deterministic non-smooth deep nets.
method De-randomized PAC-Bayes margin bounds for deterministic non-convex and non-smooth predictors.
result New bounds avoid large Lipschitz constants, providing generalization guarantees.
Safe-EF improves federated learning for non-smooth, constrained optimization.
problem Federated learning's communication bottlenecks with high-dimensional model updates.
method Error feedback (EF) for non-smooth convex optimization with safety constraints.
result Safe-EF matches lower complexity bounds and ensures safety constraints.
In machine learning research, the proximal gradient methods are popular for solving various optimization problems with non-smooth regularization. Inexact proximal gradient methods are extremely important when exactly solving the proximal operator is time-consuming, or the proximal operator does not have an analytic sol…
Difference of convex (DC) functions cover a broad family of non-convex and possibly non-smooth and non-differentiable functions, and have wide applications in machine learning and statistics. Although deterministic algorithms for DC functions have been extensively studied, stochastic optimization that is more suitable …
Paper proposes ADMM algorithms for non-smooth optimization under RDP.
problem Optimizing composite functions with non-smooth penalties under privacy constraints.
method Developed ssADMM and mpADMM algorithms for non-smooth optimization problems with RDP guarantees.
result Both ssADMM and mpADMM outperform baseline methods in high privacy settings.
New iterative regularization method tackles non-smooth, non-strongly convex functionals.
problem Tackles non-smooth, non-strongly convex functionals in regularization problems.
method Primal-dual algorithm with convergence and stability analysis.
result First iterative regularization procedure for non-smooth, non-strongly convex functionals.
A new algorithm improves both computational efficiency and statistical optimality for robust low-rank matrix and tensor estimation.
problem Challenges in low-rank matrix estimation under heavy-tailed noise, both computationally and statistically.
method Riemannian sub-gradient (RsGrad) algorithm, which is computationally efficient and statistically optimal.
result RsGrad achieves linear convergence and statistical optimality for robust loss functions under Gaussian and heavy-tailed noise.
The three operator splitting scheme was recently proposed by [Davis and Yin, 2015] as a method to optimize composite objective functions with one convex smooth term and two convex (possibly non-smooth) terms for which we have access to their proximity operator. In this short note we provide an alternative proof for the…
New algorithm solves non-convex, non-differentiable min-max games.
problem Limited theoretical understanding of non-smooth min-max games.
method Proximal gradient descent-ascent algorithm for convex-strongly convex games.
result Algorithm converges to ε-Nash equilibrium with polynomial gradient evaluations.
Stochastic gradient descent's long-term fluctuations are described by a diffusion limit.
problem Long-term behavior of stochastic gradient descent in non-smooth settings.
method Functional central limit theorem applied to rescaled trajectory of SGD.
result Characterization of long-term fluctuations around the minimizer.
FedProx algorithm improved for non-smooth and heterogeneous data.
problem Theoretical understanding of FedProx for non-convex federated optimization.
method Local dissimilarity invariant convergence theory through algorithmic stability.
result Convergence guarantees for non-smooth FL problems and minibatch size.
In this paper we study the sub-Finsler geometry as a time-optimal control problem. In particular, we consider non-smooth and non-strictly convex sub-Finsler structures associated with the Heisenberg, Grushin, and Martinet distributions. Motivated by problems in geometric group theory, we characterize extremal curves, d…
Paper analyzes convergence of stochastic methods under heavy-tailed noise.
problem Analyzing convergence of stochastic methods under heavy-tailed noise.
method Investigates vanilla and clipped stochastic subgradient descent methods.
result Demonstrates convergence properties under sub-Weibull and p-BCM noise assumptions.
Study on reproducibility in optimization with bounds on limits.
problem Limits of reproducibility in noisy or error-prone optimization procedures.
method Defined a quantitative measure of reproducibility and analyzed convex optimization settings.
result Revealed a fundamental trade-off between computation and reproducibility.
This paper presents an asynchronous incremental aggregated gradient algorithm and its implementation in a parameter server framework for solving regularized optimization problems. The algorithm can handle both general convex (possibly non-smooth) regularizers and general convex constraints. When the empirical data loss…
In this paper we study the limitations of parallelization in convex optimization. A convenient approach to study parallelization is through the prism of \emph{adaptivity} which is an information theoretic measure of the parallel runtime of an algorithm [BS18]. Informally, adaptivity is the number of sequential rounds a…
New adaptive methods for constrained convex optimization and variational inequalities.
problem Optimization of constrained convex problems and variational inequalities.
method AdaACSA and AdaAGD+ are accelerated methods that achieve nearly-optimal convergence rates for smooth and non-smooth functions.
result Achieve nearly-optimal convergence rates for both smooth and non-smooth functions, even with stochastic gradients.
Improved regret bounds for online convex optimization under stochastic and adversarial settings.
problem Interpolating between stochastic and adversarial online convex optimization.
method Optimistic online mirror descent (OMD) for the Stochastically Extended Adversarial (SEA) model.
result Established new regret bounds for various function classes.