arXiv research

Geometric step decay schedules improve stochastic algorithms' convergence on sharp nonconvex problems.

Sharp estimates for heat flow on nonconvex domains.

Sharp convergence analysis for nonconvex regression models.

A fast sketching algorithm solves regularized least squares problems efficiently.

Paper analyzes online tensorial ICA convergence with stochastic approximation.

Develops a method to estimate policy values robustly in the presence of confounding variables.

We provide theoretical analysis of the statistical and computational properties of penalized $M$-estimators that can be formulated as the solution to a possibly nonconvex optimization problem. Many important estimators fall in this category, including least squares regression with nonconvex regularization, generalized …

Sharp global guarantees for noisy overparameterized low-rank recovery.

Nonconvex matrix recovery is known to contain no spurious local minima under a restricted isometry property (RIP) with a sufficiently small RIP constant $δ$. If $δ$ is too large, however, then counterexamples containing spurious local minima are known to exist. In this paper, we introduce a proof technique that is capa…

Improved guarantees for nonconvex matrix factorization with rank overparameterization.

We provide novel theoretical results regarding local optima of regularized $M$-estimators, allowing for nonconvexity in both loss and penalty functions. Under restricted strong convexity on the loss and suitable regularity conditions on the penalty, we prove that \emph{any stationary point} of the composite objective f…

The paper analyzes how learning rate affects SGD and provides insights into optimal rates.

Laplace kernel feature selection offers statistical guarantees for nonparametric models with few samples.

PPGD solves nonconvex nonsmooth optimization problems without KL property.

We study the problem of sampling from a distribution $p^*(x) \propto \exp\left(-U(x)\right)$, where the function $U$ is $L$-smooth everywhere and $m$-strongly convex outside a ball of radius $R$, but potentially nonconvex inside this ball. We study both overdamped and underdamped Langevin MCMC and establish upper bound…

A simple function shows how neural nets can converge despite high sharpness.

Full-batch GD achieves generalization close to any stationary point with fewer assumptions.

Paper proposes an algorithm to solve complex minimax problems efficiently.

Safe reinforcement learning with nonconvex constraints using convex approximations.

Paper develops accelerated APCD for nonconvex nonsmooth problems with performance guarantees.

Two algorithms solve nonconvex minimax problems with linear constraints, achieving complexity guarantees.

New algorithms solve nonconvex-concave minimax problems without parameter knowledge.

Schedule-free SGD is optimal for nonconvex optimization problems.

This paper analyzes OGDA and EG methods for nonconvex minimax problems.

Two new algorithms solve nonconvex-strongly concave problems efficiently.

New framework explains why nonconvex methods work well in low-rank matrix estimation.

As surrogate functions of $L_0$-norm, many nonconvex penalty functions have been proposed to enhance the sparse vector recovery. It is easy to extend these nonconvex penalty functions on singular values of a matrix to enhance low-rank matrix recovery. However, different from convex optimization, solving the nonconvex l…

We study the Stochastic Gradient Descent (SGD) method in nonconvex optimization problems from the point of view of approximating diffusion processes. We prove rigorously that the diffusion process can approximate the SGD algorithm weakly using the weak form of master equation for probability evolution. In the small ste…

Algorithm recovers factors of rank-1 matrices from noisy measurements.

New algorithm solves nonconvex-convex minimax problems efficiently.

New algorithms solve complex minimax problems without needing derivatives.

AGDA and variance-reduced methods solve nonconvex-nonconcave minimax problems globally and faster.

Paper analyzes nonconvex bandit problems with improved adaptive methods.

Develops efficient method for nonconvex problems using Regula Falsi.

The use of convex regularizers allows for easy optimization, though they often produce biased estimation and inferior prediction performance. Recently, nonconvex regularizers have attracted a lot of attention and outperformed convex ones. However, the resultant optimization problem is much harder. In this paper, for a …

Survey of tractable nonconvex problems using symmetry.

New algorithm tackles nonconvex machine learning problems with adaptive normalization and independent sampling.

Adaptive gradient methods such as AdaGrad and its variants update the stepsize in stochastic gradient descent on the fly according to the gradients received along the way; such methods have gained widespread use in large-scale optimization for their ability to converge robustly, without the need to fine-tune the stepsi…

PAGE optimizes nonconvex problems with optimal convergence rates.

New method improves signal reconstruction with nonconvex penalties and parameter control.

Introduces PPMM algorithm for nonconvex robust regression problems.

We analyze stochastic algorithms for optimizing nonconvex, nonsmooth finite-sum problems, where the nonconvex part is smooth and the nonsmooth part is convex. Surprisingly, unlike the smooth case, our knowledge of this fundamental problem is very limited. For example, it is not known whether the proximal stochastic gra…

TiAda adapts adaptive gradient methods for nonconvex minimax optimization.

Gradient descent with noise converges to a unique optimum in nonconvex matrix factorization.

Paper analyzes GDA for nonconvex-concave problems, showing efficient convergence.

New algorithm for privacy-preserving nonconvex optimization.

We analyze a fast incremental aggregated gradient method for optimizing nonconvex problems of the form $\min_x \sum_i f_i(x)$. Specifically, we analyze the SAGA algorithm within an Incremental First-order Oracle framework, and show that it converges to a stationary point provably faster than both gradient descent and s…

Efficient algorithm solves sparse nonconvex regression problems.