arXiv research

MARINA-P improves non-smooth federated optimization with adaptive stepsizes.

New algorithms optimize non-smooth, non-convex objectives with improved complexity.

We consider the problem of finding local minimizers in non-convex and non-smooth optimization. Under the assumption of strict saddle points, positive results have been derived for first-order methods. We present the first known results for the non-smooth case, which requires different analysis and a different algorithm…

This work speeds up hyperparameter selection for non-smooth convex models using implicit differentiation.

Paper tackles private optimization for non-smooth objectives efficiently.

New methods improve convergence in non-convex non-smooth learning problems.

New bounds explain deterministic non-smooth deep nets without large Lipschitz constants.

Expanding FCCO to non-smooth weakly-convex problems, improving deep learning performance.

New SPS variant improves non-smooth optimization without small gradients.

The paper explores various stationarity concepts in non-smooth optimization.

New sampling algorithm for non-smooth potentials.

Optimal private ERM and SCO with subquadratic gradient complexity.

A new method solves convex optimization on curved spaces.

Advances smooth over-parameterization for solving non-smooth optimization problems.

A new optimization method, BPM, converges linearly in non-convex, non-smooth problems.

The study extends curvature bounds to non-smooth spaces and proves stability of mean curvature.

We provide improved convergence rates for various \emph{non-smooth} optimization problems via higher-order accelerated methods. In the case of $\ell_\infty$ regression, we achieves an $O(ε^{-4/5})$ iteration complexity, breaking the $O(ε^{-1})$ barrier so far present for previous methods. We arrive at a similar rate fo…

In this paper, we develop a novel {\bf ho}moto{\bf p}y {\bf s}moothing (HOPS) algorithm for solving a family of non-smooth problems that is composed of a non-smooth term with an explicit max-structure and a smooth term or a simple non-smooth term whose proximal mapping is easy to compute. The best known iteration compl…

We consider the problem of finding critical points of functions that are non-convex and non-smooth. Studying a fairly broad class of such problems, we analyze the behavior of three gradient-based methods (gradient descent, proximal update, and Frank-Wolfe update). For each of these methods, we establish rates of conver…

We prove the existence of non-smooth solutions to Special Lagrangian Equations in the non-convex case.

We investigate the theoretical limits of pipeline parallel learning of deep learning architectures, a distributed setup in which the computation is distributed per layer instead of per example. For smooth convex and non-convex objective functions, we provide matching lower and upper complexity bounds and show that a na…

We consider the problem of sampling from a density of the form $p(x) \propto \exp(-f(x)- g(x))$, where $f: \mathbb{R}^d \rightarrow \mathbb{R}$ is a smooth and strongly convex function and $g: \mathbb{R}^d \rightarrow \mathbb{R}$ is a convex and Lipschitz function. We propose a new algorithm based on the Metropolis-Has…

New index characterizes non-smooth Zoll convex bodies.

Given a convex optimization problem and its dual, there are many possible first-order algorithms. In this paper, we show the equivalence between mirror descent algorithms and algorithms generalizing the conditional gradient method. This is done through convex duality, and implies notably that for certain problems, such…

New algorithms for differentially private optimization in convex and non-convex settings with near-optimal rates.

Stochastic Gradient Descent (SGD) is one of the simplest and most popular stochastic optimization methods. While it has already been theoretically studied for decades, the classical analysis usually required non-trivial smoothness assumptions, which do not apply to many modern applications of SGD with non-smooth object…

Paper relaxes SGD privacy and generalization guarantees for non-smooth convex losses.

New iterative regularization method tackles non-smooth, non-strongly convex functionals.

New algorithm solves non-convex, non-differentiable min-max games.

This paper focuses on convex constrained optimization problems, where the solution is subject to a convex inequality constraint. In particular, we aim at challenging problems for which both projection into the constrained domain and a linear optimization under the inequality constraint are time-consuming, which render …

Stochastic gradient descent's long-term fluctuations are described by a diffusion limit.

In machine learning research, the proximal gradient methods are popular for solving various optimization problems with non-smooth regularization. Inexact proximal gradient methods are extremely important when exactly solving the proximal operator is time-consuming, or the proximal operator does not have an analytic sol…

In recent literature, a general two step procedure has been formulated for solving the problem of phase retrieval. First, a spectral technique is used to obtain a constant-error initial estimate, following which, the estimate is refined to arbitrary precision by first-order optimization of a non-convex loss function. N…

The three operator splitting scheme was recently proposed by [Davis and Yin, 2015] as a method to optimize composite objective functions with one convex smooth term and two convex (possibly non-smooth) terms for which we have access to their proximity operator. In this short note we provide an alternative proof for the…

New algorithm for robust high-dimensional linear regression is both fast and statistically optimal.

Safe-EF improves federated learning for non-smooth, constrained optimization.

New SGD covering technique yields dimension-independent generalization bounds.

In this paper, we discuss the problem of minimizing the sum of two convex functions: a smooth function plus a non-smooth function. Further, the smooth part can be expressed by the average of a large number of smooth component functions, and the non-smooth part is equipped with a simple proximal mapping. We propose a pr…

Extending Itô's formula to non-smooth functions is important both in theory and applications. One of the fairly general extensions of the formula, known as Meyer-Itô, applies to one dimensional semimartingales and convex functions. There are also satisfactory generalizations of Itô's formula for diffusion processes whe…

Unified framework for training neural networks with non-smooth, non-convex regularizers.

Predictive models can be used on high-dimensional brain images for diagnosis of a clinical condition. Spatial regularization through structured sparsity offers new perspectives in this context and reduces the risk of overfitting the model while providing interpretable neuroimaging signatures by forcing the solution to …

Paper proposes ZO-SMD for MERO, achieving optimal convergence rates.

Estimates volume of convex Alexandrov spaces with boundary.

Difference of convex (DC) functions cover a broad family of non-convex and possibly non-smooth and non-differentiable functions, and have wide applications in machine learning and statistics. Although deterministic algorithms for DC functions have been extensively studied, stochastic optimization that is more suitable …

This work aims at recovering signals that are sparse on graphs. Compressed sensing offers techniques for signal recovery from a few linear measurements and graph Fourier analysis provides a signal representation on graph. In this paper, we leverage these two frameworks to introduce a new Lasso recovery algorithm on gra…

We analyze convergence rates of stochastic optimization procedures for non-smooth convex optimization problems. By combining randomized smoothing techniques with accelerated gradient methods, we obtain convergence rates of stochastic optimization procedures, both in expectation and with high probability, that have opti…

We propose inertial versions of block coordinate descent methods for solving non-convex non-smooth composite optimization problems. Our methods possess three main advantages compared to current state-of-the-art accelerated first-order methods: (1) they allow using two different extrapolation points to evaluate the grad…

A new algorithm speeds up sparse-penalized quantile regression solving non-convex penalties.