New nonconvex penalty smooths at origin for deep learning.
problem Improving variable selection and bias in high-dimensional statistical learning.
method Developed a new nonconvex penalty function smooth at origin.
result Asymptotic bias of new penalty function vanishes exponentially fast.
New framework explains why nonconvex methods work well in low-rank matrix estimation.
problem Nonconvex low-rank matrix estimation problems in machine learning.
method Developed a theoretical framework revealing a benign regularizer.
result Nonconvex procedures can behave well due to a disguised convexity.
The use of convex regularizers allows for easy optimization, though they often produce biased estimation and inferior prediction performance. Recently, nonconvex regularizers have attracted a lot of attention and outperformed convex ones. However, the resultant optimization problem is much harder. In this paper, for a …
New algorithm tackles nonconvex machine learning problems with adaptive normalization and independent sampling.
problem Nonconvex machine learning problems with generalized-smoothness.
method Adaptive gradient normalization, independent sampling, and gradient clipping.
result Achieves an O(ε^(-4)) sample complexity for fast convergence.
Schedule-free SGD is optimal for nonconvex optimization problems.
problem Nonconvex optimization in neural networks.
method Developed a general framework for online-to-nonconvex conversion, which converts schedule-free SGD into an effective nonconvex optimization algorithm.
result Schedule-free SGD achieves optimal iteration complexity for nonsmooth, nonconvex optimization problems.
We consider an online learning process to forecast a sequence of outcomes for nonconvex models. A typical measure to evaluate online learning algorithms is regret but such standard definition of regret is intractable for nonconvex models even in offline settings. Hence, gradient based definition of regrets are common f…
PPGD solves nonconvex nonsmooth optimization problems without KL property.
problem Nonconvex and nonsmooth optimization problems in statistics and machine learning.
method Projective Proximal Gradient Descent (PPGD) for solving a class of nonconvex and nonsmooth problems.
result PPGD achieves a fast convergence rate of O(1/k^2) for k ≥ k_0.
Stochastic gradient descent (SGD) is a popular and efficient method with wide applications in training deep neural nets and other nonconvex models. While the behavior of SGD is well understood in the convex learning setting, the existing theoretical results for SGD applied to nonconvex objective functions are far from …
AGDA and variance-reduced methods solve nonconvex-nonconcave minimax problems globally and faster.
problem Solving nonconvex-nonconcave minimax problems in machine learning.
method Global convergence of AGDA and variance-reduced algorithms.
result AGDA and variance-reduced methods achieve global convergence and faster rates.
Efficient solver for nonconvex tensor regularization reduces computational cost.
problem Computational inefficiency in extending nonconvex regularization to tensor learning.
method Proximal average algorithm with adaptive momentum, maintaining sparse plus low-rank structure.
result Shows good statistical performance and accuracy on tensor completion problems.
Paper develops algorithms for nonsmooth, nonconvex statistical learning problems.
problem Nonsmooth and nonconvex objectives in statistical learning.
method Bregman-surrogate algorithm framework, including local linear approximation, mirror descent, iterative thresholding, DC programming.
result Global convergence rates for nonconvex and nonsmooth objectives in high dimensions.
Paper proposes a new method for training nonconvex models.
problem Training nonconvex models like neural networks.
method Successive functional gradient optimization using mirror descent in a function space.
result The method leads to better performance than standard training techniques.
New algorithms solve nonconvex federated learning problems efficiently.
problem Nonconvex federated composite optimization in federated learning.
method FedDR and asyncFedDR algorithms combining Douglas-Rachford splitting, randomized block-coordinate strategies, and asynchronous implementation.
result Match communication complexity lower bound up to a constant factor.
Accelerated gradient method tackles nonconvex penalties in sparse learning.
problem Optimizing nonconvex penalties in sparse statistical learning.
method Generalized Nesterov's accelerated gradient method with hyperparameter optimization.
result Convergence can be made considerably faster with optimal hyperparameters.
Unified parametric assumption improves convergence guarantees for nonconvex optimization.
problem Weak convergence guarantees for nonconvex optimization.
method Introducing a novel unified parametric assumption.
result Unified convergence theorem for gradient-based methods.
Paper proposes an algorithm to solve complex minimax problems efficiently.
problem Stochastic nonconvex-concave minimax problems in various fields.
method Accelerated first-order regularized momentum descent ascent algorithm (FORMDA).
result Achieves best-known complexity bound of ildeO(ε−6.5) for single-loop algorithms. The paper analyzes PPM for nonconvex-nonconcave problems, identifying three regions with varying convergence guarantees.
problem Challenges in nonconvex-nonconcave minimax optimization.
method Classic proximal point method with insights from the Moreau envelope.
result Identification of three regions with varying convergence guarantees for PPM.
New method reduces communication costs in distributed nonconvex optimization.
problem Large communication costs between central server and local workers in distributed learning.
method Communication-compressed AMSGrad for distributed nonconvex optimization.
result Converges to first-order stationary point with same iteration complexity as vanilla AMSGrad.
In the past decade, sparse and low-rank recovery have drawn much attention in many areas such as signal/image processing, statistics, bioinformatics and machine learning. To achieve sparsity and/or low-rankness inducing, the ℓ1 norm and nuclear norm are of the most popular regularization penalties due to their co…
Paper proposes a faster SPIDER-EM variant for large-scale nonconvex optimization.
problem High computational cost of EM algorithm in large-scale learning.
method Extension of SPIDER-EM for nonconvex finite-sum optimization problems.
result Achieves state-of-the-art complexity bounds and linear convergence under certain conditions.
With the large rising of complex data, the nonconvex models such as nonconvex loss function and nonconvex regularizer are widely used in machine learning and pattern recognition. In this paper, we propose a class of mini-batch stochastic ADMMs (alternating direction method of multipliers) for solving large-scale noncon…
Regularization methods are often employed in deep learning neural networks (DNNs) to prevent overfitting. For penalty based DNN regularization methods, convex penalties are typically considered because of their optimization guarantees. Recent theoretical work have shown that nonconvex penalties that satisfy certain reg…
We study Frank-Wolfe methods for nonconvex stochastic and finite-sum optimization problems. Frank-Wolfe methods (in the convex case) have gained tremendous recent interest in machine learning and optimization communities due to their projection-free property and their ability to exploit structured constraints. However,…
New algorithm solves nonconvex-convex minimax problems efficiently.
problem Solving nonconvex-convex minimax problems with nonsmooth, nonconvex, and nonlinearity.
method Hybrid variance-reduced SGD algorithm combining smoothing and biased techniques.
result Achieves O(T^(-2/3)) convergence rate and best oracle complexity.
Paper reviews advances in solving sparsest vector problem in subspaces.
problem Finding the sparsest vector in a low-dimensional subspace.
method Geometric analysis of optimization landscapes and efficient nonconvex optimization algorithms.
result Recent advances in global nonconvex optimization for sparsest vector problem.
We study the safe reinforcement learning problem with nonlinear function approximation, where policy optimization is formulated as a constrained optimization problem with both the objective and the constraint being nonconvex functions. For such a problem, we construct a sequence of surrogate convex constrained optimiza…
Unified framework for constructing nonconvex sparse recovery methods.
problem Constructing valid nonconvex regularization functions remains open.
method Unified framework based on probability density function, using Weibull distribution.
result New nonconvex sparse recovery method based on Weibull distribution.
Two algorithms solve nonconvex minimax problems with linear constraints, achieving complexity guarantees.
problem Nonconvex minimax problems with coupled linear constraints.
method Zeroth-order primal-dual alternating projected gradient (ZO-PDAPG) and zeroth-order regularized momentum primal-dual projected gradient (ZO-RMPDPG) algorithms.
result Iteration complexity guarantees for solving nonconvex-(strongly) concave minimax problems with coupled linear constraints.
The problem of minimizing sum-of-nonconvex functions (i.e., convex functions that are average of non-convex ones) is becoming increasingly important in machine learning, and is the core machinery for PCA, SVD, regularized Newton's method, accelerated non-convex optimization, and more. We show how to provably obtain an …
TiAda adapts adaptive gradient methods for nonconvex minimax optimization.
problem Nonconvex minimax optimization challenges in achieving convergence.
method TiAda is a time-scale adaptive GDA algorithm for nonconvex minimax optimization.
result TiAda achieves near-optimal complexities in deterministic and stochastic settings.
Two-Timescale EM Methods improve EM for nonconvex models.
problem Nonconvex latent variable models are challenging for EM.
method Two-stage stochastic updates to handle nonconvex optimization.
result Global convergence for nonconvex objective functions.
We provide theoretical analysis of the statistical and computational properties of penalized M-estimators that can be formulated as the solution to a possibly nonconvex optimization problem. Many important estimators fall in this category, including least squares regression with nonconvex regularization, generalized …
Optimization geometry affects deep learning performance.
problem The impact of optimization geometry on deep learning performance.
method Analysis of pseudogradient methods for learning generalized linear models.
result Non-asymptotic bounds on generalization error characterize model performance.
Studied SGD convergence under weak conditions.
problem Convergence of SGD in nonconvex optimization.
method Analyzed biased nonconvex SGD under mild conditions.
result Provided convergence rates and complexities.
Study optimization landscapes for overcomplete representations, showing benign geometric structures.
problem Optimizing overcomplete representations in high-dimensional data analysis.
method Formulate as ℓ4-norm optimization problems with spherical constraint, analyze geometric properties. result Nonconvex objectives have benign geometric structures, ensuring local search algorithms find target solutions.
A fast sketching algorithm solves regularized least squares problems efficiently.
problem Solving large-scale optimization problems with convex or nonconvex regularization.
method Sketching for Regularized Optimization (SRO) algorithm that generates a sketch of the original data matrix and solves the sketched problem.
result General theoretical results for the approximation error between the original and sketched problems, including minimax rates for sparse signal estimation.
Matrix completion has attracted much interest in the past decade in machine learning and computer vision. For low-rank promotion in matrix completion, the nuclear norm penalty is convenient due to its convexity but has a bias problem. Recently, various algorithms using nonconvex penalties have been proposed, among whic…
Study efficient algorithms for nonconvex optimization with state-dependent Markov data.
problem Stochastic optimization with Markovian data and state-dependent transition kernels.
method Projection-based and projection-free algorithms for constrained nonconvex problems.
result The number of oracle calls to achieve an ε-stationary point is O(1/ε2.5). Improved SGD methods converge faster for nonconvex optimization.
problem Nonconvex optimization challenges in machine learning.
method Adaptive SGD with line-search and Polyak stepsizes.
result Unified convergence rates for various nonconvex functions.
High-dimensional data pose challenges in statistical learning and modeling. Sometimes the predictors can be naturally grouped where pursuing the between-group sparsity is desired. Collinearity may occur in real-world high-dimensional applications where the popular l1 technique suffers from both selection inconsisten…
The paper analyzes how learning rate affects SGD and provides insights into optimal rates.
problem Understanding the impact of learning rate on stochastic gradient descent.
method Developed a learning-rate-dependent stochastic differential equation (lr-dependent SDE) to analyze SGD.
result Established a linear rate of convergence for SGD and found the optimal linear rate by analyzing the spectrum of the Witten-Laplacian.
New algorithms solve complex minimax problems without needing derivatives.
problem Solving nonconvex-concave minimax problems efficiently.
method Zeroth-order alternating and proximal gradient algorithms.
result Iteration complexity and function value estimation bounds established.
Improved complexity for smooth nonconvex optimization using quasi-Newton methods.
problem Finding ε-first-order stationary points of smooth functions with gradient information only.
method Two-level online learning approach involving quasi-Newton methods.
result Gradient complexity improved to O(d^(1/4)ε^(-13/8)) for d = O(ε^(-1/2)).
Adaptive gradient methods are workhorses in deep learning. However, the convergence guarantees of adaptive gradient methods for nonconvex optimization have not been thoroughly studied. In this paper, we provide a fine-grained convergence analysis for a general class of adaptive gradient methods including AMSGrad, RMSPr…
New algorithm for privacy-preserving nonconvex optimization.
problem Privacy-preserving nonconvex empirical risk minimization.
method Differentially private stochastic gradient descent algorithm.
result Achieves strong privacy guarantees efficiently with improved utility.
RELTA-SGLD stabilizes nonconvex SGLD updates with a lighter taming scheme.
problem Stabilizing superlinear stochastic-gradient updates in nonconvex optimization.
method Threshold-based taming with relative-growth principle for stability.
result Polynomial moment stability and first-order stationary accuracy in nonconvex SGLD.
Unified Lagrangian-based methods for nonsmooth nonconvex optimization.
problem Minimizing nonsmooth nonconvex functions with constraints.
method Developed a unified framework for Lagrangian-based methods using subgradient updates.
result Global convergence guarantees for the proposed framework under mild conditions.
MAML optimizes shared priors for subtasks in a nonconvex meta-objective.
problem Understanding global optimality of MAML for nonconvex meta-objectives.
method Characterizes optimality gap of MAML stationary points via first-order optimization methods.
result Establishes global optimality of MAML for both RL and supervised learning.