Unified parametric assumption improves convergence guarantees for nonconvex optimization.
problem Weak convergence guarantees for nonconvex optimization.
method Introducing a novel unified parametric assumption.
result Unified convergence theorem for gradient-based methods.
Unified framework for nonconvex matrix completion with linearly parameterized factors.
problem Matrix completion with improved accuracy using linearly parameterized factors.
method Unified nonconvex optimization framework with Correlated Parametric Factorization condition.
result Uniform upper bounds for low-rank estimation at any local minimum.
Geometric insights improve convergence of implicit generative models.
problem Improving convergence of implicit generative models.
method Analyzing geometries induced by Wasserstein distance and other criteria.
result Established surprising approximate global convergence guarantees for the 1-Wasserstein distance.
This paper shows linear over-parametrization suffices for shallow neural networks to fit training data.
problem Training shallow neural networks with optimal over-parametrization.
method Used a simple variant of stochastic gradient descent.
result Linear over-parametrization is sufficient for shallow neural networks to fit training data.
New method reduces over-parametrization in neural networks, ensuring sparsity and finite network size.
problem Over-parametrization leads to too many active neurons in neural networks, especially with large data.
method Investigates a nonconvex regularization method for shallow ReLU networks.
result Locally optimal networks are finite even with infinite data, maintaining approximation guarantees and network size bounds.
Proposes a generalized XGBoost method for nonconvex loss functions.
problem Limited to convex loss functions in XGBoost.
method Extends XGBoost to use nonconvex loss functions and multivariate loss functions.
result Generalized XGBoost method can model multiple parameters in various distributions.
Improved zeroth-order algorithms tackle nonconvex minimax problems with reduced complexity.
problem Nonconvex minimax optimization problems in machine learning.
method Design and analysis of Zeroth-Order Gradient Descent Ascent ( exttt{ZO-GDA}) and Zeroth-Order Gradient Descent Multi-Step Ascent ( exttt{ZO-GDMSA}) algorithms.
result Oracle complexity improvements for minimax optimization problems.
Study on different regularization methods in remote sensing.
problem High dimensional image classification and sparse linear unmixing.
method Traditional squared and sparsity-promoting norms, as well as nonconvex regularizers (p and Log Sum Penalty) are compared.
result Advantages of different regularization methods are provided for various tasks.
New method solves complex optimization problems with real-time learning.
problem Nonconvex nonsmooth conditional stochastic optimization problems.
method Single time-scale stochastic method with parametric model approximation.
result Method converges with probability one using differential inclusions and Lyapunov function.
The paper optimizes bridge-type estimators for sparse models using pathwise methods.
problem Sparse parametric models with adaptive coefficients and multiple penalties.
method Pathwise optimization with accelerated proximal gradient descent and blockwise alternating optimization.
result Efficient computation of the full solution path for adaptive bridge estimators.
Unified framework for constructing nonconvex sparse recovery methods.
problem Constructing valid nonconvex regularization functions remains open.
method Unified framework based on probability density function, using Weibull distribution.
result New nonconvex sparse recovery method based on Weibull distribution.
New method improves signal reconstruction with nonconvex penalties and parameter control.
problem Reconstructing sparse signals with nonconvex penalties and nonconvexity control.
method Introduces nonconvex penalties (SCAD, MCP) with nonconvexity parameters and controls them to guide AMP trajectory.
result Achieves perfect reconstruction for relatively dense signals with small nonconvexity parameters.
We study Langevin diffusion for nonconvex functions with manifold structure.
problem Sampling from distributions with nonconvex functions and manifolds of equal probability.
method Prove mixing time bounds for Langevin diffusion using manifold geometry, specialize to matrix factorization problems.
result Langevin diffusion mixes rapidly on manifolds of equal probability in nonconvex functions.
Improved SGD methods converge faster for nonconvex optimization.
problem Nonconvex optimization challenges in machine learning.
method Adaptive SGD with line-search and Polyak stepsizes.
result Unified convergence rates for various nonconvex functions.
Schedule-free SGD is optimal for nonconvex optimization problems.
problem Nonconvex optimization in neural networks.
method Developed a general framework for online-to-nonconvex conversion, which converts schedule-free SGD into an effective nonconvex optimization algorithm.
result Schedule-free SGD achieves optimal iteration complexity for nonsmooth, nonconvex optimization problems.
Paper analyzes SARAH for nonconvex optimization with mini-batches.
problem Solving nonconvex optimization problems with mini-batches.
method Stochastic Recursive Gradient Algorithm (SARAH) for nonconvex losses.
result Sublinear and linear convergence rates for different types of nonconvex functions.
New algorithm for privacy-preserving nonconvex optimization.
problem Privacy-preserving nonconvex empirical risk minimization.
method Differentially private stochastic gradient descent algorithm.
result Achieves strong privacy guarantees efficiently with improved utility.
The paper introduces ADMM methods with variance reduction for nonconvex optimization.
problem Nonconvex optimization problems.
method Stochastic ADMM with variance reduction.
result Iteration complexity bound of O(1/ε) for obtaining an ε-stationary solution. Efficient ADMM algorithm solves nonconvex SVMs with various penalties.
problem Solving nonconvex penalized SVMs due to nondifferentiability, nonsmoothness, and nonconvexity.
method ADMM-based algorithm for a wide range of nonconvex penalties.
result The proposed algorithm outperforms other methods on benchmark datasets.
PPGD solves nonconvex nonsmooth optimization problems without KL property.
problem Nonconvex and nonsmooth optimization problems in statistics and machine learning.
method Projective Proximal Gradient Descent (PPGD) for solving a class of nonconvex and nonsmooth problems.
result PPGD achieves a fast convergence rate of O(1/k^2) for k ≥ k_0.
New algorithms solve nonconvex, nonsmooth optimization problems.
problem Optimizing nonconvex, nonsmooth finite-sum problems with limited existing knowledge.
method Developed fast stochastic algorithms for constant minibatches.
result Proved global linear convergence rate for a specific class of functions.
New method solves doubly-nonconvex composite optimization problems.
problem Solving composite optimization problems with both functions nonconvex.
method Stochastic gradient descent with quasiconvex penalty function.
result Convergence properties for doubly-nonconvex composite optimization.
Paper proposes a method to make nonconvex optimization more efficient.
problem Nonconvex regularizers improve performance but are harder to optimize.
method Redistribute nonconvexity from regularizer to loss, making it convex.
result Optimization with convexified regularizer is faster and more efficient.
New framework explains why nonconvex methods work well in low-rank matrix estimation.
problem Nonconvex low-rank matrix estimation problems in machine learning.
method Developed a theoretical framework revealing a benign regularizer.
result Nonconvex procedures can behave well due to a disguised convexity.
Develops shuffling gradient-based methods for nonconvex-concave minimax optimization.
problem Nonconvex-concave minimax optimization problems.
method Two shuffling gradient-based algorithms for nonconvex-linear and nonconvex-strongly concave settings.
result Achieves state-of-the-art oracle complexity in nonconvex optimization and best-known complexity bounds for nonconvex-strongly concave setting.
New method estimates large covariance matrices using nonconvex penalties.
problem Estimating large covariance matrices in high-dimensional data.
method Developed a first-order algorithm using generalized nonconvex penalties.
result Positive-definite covariance estimators using nonconvex penalties.
Paper proposes mini-batch ADMMs for solving nonconvex nonsmooth optimization problems.
problem Solving large-scale nonconvex nonsmooth optimization problems.
method Proposes mini-batch stochastic ADMMs for nonconvex nonsmooth optimization.
result Mini-batch stochastic ADMMs converge to a stationary point with rate O(1/T).
New nonconvex Frank-Wolfe methods for faster optimization.
problem Nonconvex optimization problems in machine learning.
method Stochastic Frank-Wolfe methods for nonconvex optimization.
result Improved convergence rates for nonconvex optimization.
New definition of regret for nonconvex online learning models.
problem Intractability of standard regret measures for nonconvex models.
method Introduced a local gradient based regret definition.
result Our definition provides more interpretable bounds for forecasting.
Study nonconvex matrix completion for low-rank approximation without rank assumptions.
problem Low-rank approximation of positive semidefinite matrices from partial entries.
method Nonconvex optimization, local-minimum analysis, no spurious local minima.
result Improved sampling rate for nonconvex matrix completion with no spurious local minima.
SVRG accelerates nonconvex optimization faster than SGD and GD.
problem Nonconvex optimization problems.
method Stochastic variance reduced gradient (SVRG) methods.
result SVRG converges faster to stationary points and global optima than SGD and GD.
Paper analyzes fast SAGA method for nonconvex optimization problems.
problem Optimizing nonconvex problems of the form minx∑ifi(x) method Incremental aggregated gradient method (SAGA) within an Incremental First-order Oracle framework
result SAGA converges to a stationary point faster than gradient descent and stochastic gradient descent, and at a linear rate to the global optimum for a specific class of nonconvex problems.
This paper analyzes OGDA and EG methods for nonconvex minimax problems.
problem Theoretical guarantees of OGDA and EG methods in nonconvex settings.
method Unified analysis through single-call extra-gradient methods.
result Established convergence of OGDA and EG methods under NC-SC and NC-C settings.
Simple DP algorithms find approximate solutions for nonconvex ERM.
problem Finding approximate solutions to nonconvex ERM problems with privacy.
method Differential privacy, descent directions, line search, mini-batching, two-phase strategy.
result Effective algorithms for nonconvex ERM with privacy guarantees.
The paper explores nonconvex penalties for deep learning regularization.
problem Overfitting in deep learning neural networks.
method Examines and evaluates nonconvex penalties for DNN regularization.
result Nonconvex penalties, under certain conditions, can perform well in DNNs.
NESTT tackles nonconvex optimization problems in a distributed and stochastic manner.
problem Nonconvex optimization problems with a sum of nonconvex functions and a nonsmooth regularizer.
method NESTT algorithm that splits the problem into N subproblems and uses an augmented Lagrangian based primal-dual scheme.
result NESTT achieves ε-stationary solution using O((\sum_{i=1}^N\sqrt{L_i/N})^2/ε) gradient evaluations, up to N times better than gradient descent methods.
PGD algorithm converges to local minima in nonconvex matrix completion.
problem Matrix completion with low-rank promotion using nonconvex penalties.
method Proximal gradient descent algorithm for nonconvex penalties.
result PGD algorithm converges to restricted strictly local minimizers with eventually linear rate.
Paper proposes a nonconvex approach for sparse reduced rank regression.
problem Sparse reduced rank regression model estimation problem.
method Formulated as a nonconvex optimization problem with alternating minimization method.
result Nonconvex function leads to better estimation accuracy and efficiency.
Paper removes bounded gradient assumption for SGD in nonconvex learning.
problem Existing theoretical results for SGD in nonconvex learning require uniform boundedness of gradients, which is hard to verify.
method Establishes sufficient conditions for SGD convergence without bounded gradient assumption.
result SGD achieves optimal convergence rates for nonconvex and gradient-dominated objectives.
This paper establishes lower bounds for smooth nonconvex finite-sum optimization.
problem Understanding the complexity of finding optimal solutions in nonconvex finite-sum optimization.
method Proving tight lower bounds for the complexity of finding ε-suboptimal points and ε-approximate stationary points.
result Existing algorithms achieve optimal IFO complexity up to logarithmic factors.
Paper analyzes convergence of adaptive gradient methods for nonconvex optimization.
problem Lack of convergence guarantees for adaptive gradient methods in nonconvex optimization.
method Fine-grained convergence analysis of adaptive gradient methods including AMSGrad, RMSProp, and AdaGrad.
result Proves adaptive gradient methods converge to first-order stationary points for smooth nonconvex functions.
New algorithm tackles nonconvex machine learning problems with adaptive normalization and independent sampling.
problem Nonconvex machine learning problems with generalized-smoothness.
method Adaptive gradient normalization, independent sampling, and gradient clipping.
result Achieves an O(ε^(-4)) sample complexity for fast convergence.
As surrogate functions of L0-norm, many nonconvex penalty functions have been proposed to enhance the sparse vector recovery. It is easy to extend these nonconvex penalty functions on singular values of a matrix to enhance low-rank matrix recovery. However, different from convex optimization, solving the nonconvex l…
Accelerated SGD method converges linearly to local minima of nonconvex problems.
problem Nonconvex nonsmooth optimization problems.
method Combining variance reduction and Nesterov's extrapolation for accelerated SGD.
result Linear convergence to a stationary point of the nonconvex optimization problem.
Two algorithms solve nonconvex minimax problems with linear constraints, achieving complexity guarantees.
problem Nonconvex minimax problems with coupled linear constraints.
method Zeroth-order primal-dual alternating projected gradient (ZO-PDAPG) and zeroth-order regularized momentum primal-dual projected gradient (ZO-RMPDPG) algorithms.
result Iteration complexity guarantees for solving nonconvex-(strongly) concave minimax problems with coupled linear constraints.
Paper proposes a new method for training nonconvex models.
problem Training nonconvex models like neural networks.
method Successive functional gradient optimization using mirror descent in a function space.
result The method leads to better performance than standard training techniques.
Paper proposes an algorithm to solve complex minimax problems efficiently.
problem Stochastic nonconvex-concave minimax problems in various fields.
method Accelerated first-order regularized momentum descent ascent algorithm (FORMDA).
result Achieves best-known complexity bound of ildeO(ε−6.5) for single-loop algorithms. New algorithm solves nonconvex-convex minimax problems efficiently.
problem Solving nonconvex-convex minimax problems with nonsmooth, nonconvex, and nonlinearity.
method Hybrid variance-reduced SGD algorithm combining smoothing and biased techniques.
result Achieves O(T^(-2/3)) convergence rate and best oracle complexity.