arXiv research

Develops shuffling gradient-based methods for nonconvex-concave minimax optimization.

Develops efficient method for nonconvex problems using Regula Falsi.

Two algorithms solve nonconvex minimax problems with linear constraints, achieving complexity guarantees.

Improved SGD methods converge faster for nonconvex optimization.

This paper analyzes OGDA and EG methods for nonconvex minimax problems.

Schedule-free SGD is optimal for nonconvex optimization problems.

We consider compressed sensing formulated as a minimization problem of nonconvex sparse penalties, Smoothly Clipped Absolute deviation (SCAD) and Minimax Concave Penalty (MCP). The nonconvexity of these penalties is controlled by nonconvexity parameters, and L1 penalty is contained as a limit with respect to these para…

We study Frank-Wolfe methods for nonconvex stochastic and finite-sum optimization problems. Frank-Wolfe methods (in the convex case) have gained tremendous recent interest in machine learning and optimization communities due to their projection-free property and their ability to exploit structured constraints. However,…

Smooth finite-sum optimization has been widely studied in both convex and nonconvex settings. However, existing lower bounds for finite-sum optimization are mostly limited to the setting where each component function is (strongly) convex, while the lower bounds for nonconvex finite-sum optimization remain largely unsol…

While many solutions for privacy-preserving convex empirical risk minimization (ERM) have been developed, privacy-preserving nonconvex ERM remains a challenge. We study nonconvex ERM, which takes the form of minimizing a finite-sum of nonconvex loss functions over a training set. We propose a new differentially private…

PAGE optimizes nonconvex problems with optimal convergence rates.

Unified framework for constrained diffusion models on nonconvex sets with efficient landing mechanism.

New framework explains why nonconvex methods work well in low-rank matrix estimation.

Paper analyzes nonconvex bandit problems with improved adaptive methods.

Paper develops methods for statistical inference with SGD in nonconvex optimization.

Unified parametric assumption improves convergence guarantees for nonconvex optimization.

As surrogate functions of $L_0$-norm, many nonconvex penalty functions have been proposed to enhance the sparse vector recovery. It is easy to extend these nonconvex penalty functions on singular values of a matrix to enhance low-rank matrix recovery. However, different from convex optimization, solving the nonconvex l…

AGDA and variance-reduced methods solve nonconvex-nonconcave minimax problems globally and faster.

We consider an online learning process to forecast a sequence of outcomes for nonconvex models. A typical measure to evaluate online learning algorithms is regret but such standard definition of regret is intractable for nonconvex models even in offline settings. Hence, gradient based definition of regrets are common f…

Bandit algorithms have been predominantly analyzed in the convex setting with function-value based stationary regret as the performance measure. In this paper, motivated by online reinforcement learning problems, we propose and analyze bandit algorithms for both general and structured nonconvex problems with nonstation…

Accelerated gradient method tackles nonconvex penalties in sparse learning.

We study nonconvex finite-sum problems and analyze stochastic variance reduced gradient (SVRG) methods for them. SVRG and related methods have recently surged into prominence for convex optimization given their edge over stochastic gradient descent (SGD); but their theoretical analysis almost exclusively assumes convex…

We consider nonconvex-concave minimax problems, $\min_{\mathbf{x}} \max_{\mathbf{y} \in \mathcal{Y}} f(\mathbf{x}, \mathbf{y})$, where $f$ is nonconvex in $\mathbf{x}$ but concave in $\mathbf{y}$ and $\mathcal{Y}$ is a convex and bounded set. One of the most popular algorithms for solving this problem is the celebrated…

Paper analyzes complexity of solving nonconvex-strongly-concave problems.

TiAda adapts adaptive gradient methods for nonconvex minimax optimization.

Stochastic gradient descent (SGD) is a popular and efficient method with wide applications in training deep neural nets and other nonconvex models. While the behavior of SGD is well understood in the convex learning setting, the existing theoretical results for SGD applied to nonconvex objective functions are far from …

Paper proposes a faster SPIDER-EM variant for large-scale nonconvex optimization.

We demonstrate that the primal-dual witness proof method may be used to establish variable selection consistency and $\ell_\infty$-bounds for sparse regression problems, even when the loss function and/or regularizer are nonconvex. Using this method, we derive two theorems concerning support recovery and $\ell_\infty$-…

NeAda solves nonconvex minimax optimization by balancing primal and dual variables adaptively.

This paper studies first order methods for solving smooth minimax optimization problems $\min_x \max_y g(x,y)$ where $g(\cdot,\cdot)$ is smooth and $g(x,\cdot)$ is concave for each $x$. In terms of $g(\cdot,y)$, we consider two settings -- strongly convex and nonconvex -- and improve upon the best known rates in both. …

Efficient solver for nonconvex tensor regularization reduces computational cost.

The paper analyzes PPM for nonconvex-nonconcave problems, identifying three regions with varying convergence guarantees.

Symmetric nonnegative matrix factorization (SymNMF) has important applications in data analytics problems such as document clustering, community detection and image segmentation. In this paper, we propose a novel nonconvex variable splitting method for solving SymNMF. The proposed algorithm is guaranteed to converge to…

New lower bounds for bilevel optimization with first-order oracles.

We adapt the Douglas-Rachford (DR) splitting method to solve nonconvex feasibility problems by studying this method for a class of nonconvex optimization problem. While the convergence properties of the method for convex problems have been well studied, far less is known in the nonconvex setting. In this paper, for the…

We analyze the performance of alternating minimization for loss functions optimized over two variables, where each variable may be restricted to lie in some potentially nonconvex constraint set. This type of setting arises naturally in high-dimensional statistics and signal processing, where the variables often reflect…

New algorithm solves structured nonconvex-nonconcave min-max problems.

We study functions whose truncations are convex or quasiconvex.

Unified analysis for shuffling-type gradient methods in optimization.

New algorithm solves minimax games with linear constraints.

Nonconvex and nonsmooth problems have recently attracted considerable attention in machine learning. However, developing efficient methods for the nonconvex and nonsmooth optimization problems with certain performance guarantee remains a challenge. Proximal coordinate descent (PCD) has been widely used for solving opti…

Unified framework for constructing nonconvex sparse recovery methods.

SGLD proves geometric ergodicity via reflection coupling for nonconvex log-concave distributions.

Continuous optimization is an important problem in many areas of AI, including vision, robotics, probabilistic inference, and machine learning. Unfortunately, most real-world optimization problems are nonconvex, causing standard convex techniques to find only local optima, even with extensions like random restarts and …

New method reduces communication costs in distributed nonconvex optimization.

OLLA framework efficiently samples from constrained distributions with nonconvex constraints.

In this paper, we study a nonconvex continuous relaxation of MAP inference in discrete Markov random fields (MRFs). We show that for arbitrary MRFs, this relaxation is tight, and a discrete stationary point of it can be easily reached by a simple block coordinate descent algorithm. In addition, we study the resolution …

New algorithm for nonconvex optimization on constrained Riemannian manifolds converges quickly.