New algorithm improves convergence for non-convex problems with boundaries.
problem Optimizing non-convex problems with constraints.
method Reflected Gradient Langevin Dynamics with probabilistic representation.
result Promising convergence rates, faster than existing methods.
The paper proves LF-PSGD convergence for non-convex problems.
problem Optimizing non-convex problems efficiently in parallel.
method Developed and analyzed two LF-PSGD methods: Hogwild! and AsySVRG.
result Theoretical proof of convergence for non-convex problems.
First order methods can take extremely long to find global minima of non-convex functions.
problem Finding global minimizers of non-convex functions.
method Designing a family of non-convex functions and using statistical lower bounds for parameter estimation.
result First order methods can take exponential time to converge to a global minimizer.
New algorithm solves non-convex min-max problems in signal processing.
problem Non-convex min-max problems in signal processing and communication.
method Hybrid Block Successive Approximation (HiBSA) algorithm alternating gradient descent and ascent steps.
result HiBSA converges to first-order stationary solutions with global rates.
New algorithm finds local minima in non-convex, non-smooth problems.
problem Finding local minimizers in non-convex and non-smooth optimization.
method Perturbed Proximal Descent, tailored for non-smooth cases.
result First known results for non-smooth optimization.
This monograph explores non-convex optimization techniques for machine learning.
problem Capturing complex learning and prediction problems with non-convex optimization.
method Analyzes various non-convex optimization techniques and their applications.
result Direct non-convex optimization methods often outperform relaxation-based techniques.
Quantum annealing outperforms classical in solving non-convex optimization problems crucial for machine learning.
problem Solving non-convex optimization problems efficiently.
method Designing a classical energy function and adding a quantum transverse field to facilitate tunneling.
result Quantum annealing converges efficiently to optimal solutions in a wide class of non-convex problems, unlike classical thermal annealing.
Optimizers find approximate global minima in non-convex problems.
problem Understanding why local methods solve non-convex optimization problems.
method Formalizing the hypothesis that many local minima are approximately global minima.
result Most local minima of practical non-convex objectives are approximately global minima.
Non-convex sparsity-inducing penalties have recently received considerable attentions in sparse learning. Recent theoretical investigations have demonstrated their superiority over the convex counterparts in several sparse learning settings. However, solving the non-convex optimization problems associated with non-conv…
SGD's uncertainty quantified in non-convex learning problems.
problem Uncertainty quantification in non-convex learning problems.
method Asymptotic normality of SGD iterates and bias characterization.
result SGD iterates are asymptotically normally distributed around the expected value of the invariant distribution.
Paper uses integer programming for non-convex boosting in classification.
problem Improving classification performance using non-convex optimization.
method Non-convex boosting via integer programming.
result Results comparable to or better than state-of-the-art.
Adaptive momentum method solves non-convex min-max problems.
problem Non-convex min-max optimization problems in training generative adversarial networks.
method Proposes an adaptive momentum algorithm for non-convex min-max optimization.
result Establishes non-asymptotic convergence rates for the proposed algorithm.
Paper solves curvature equations in Minkowski space for non-convex domains.
problem Solving curvature equations in non-convex domains of Minkowski space.
method Existence theorem proved via \emph{a priori} estimates and Serrin-type condition.
result Existence of solutions for curvature equations in non-convex domains.
Paper tackles non-convex inf-projection problems with stochastic optimization.
problem Non-convex and possibly non-smooth inf-projection minimization problems.
method Developed stochastic algorithms for finding (nearly) stationary solutions.
result Established first-order convergence for non-convex inf-projection problems.
Heavy Ball method speeds up finding global optima in non-convex problems.
problem Finding global optima in non-convex optimization problems.
method Heavy Ball momentum in non-convex optimization.
result Heavy Ball helps iterates enter a benign region faster, containing a global optimal point.
New algorithm tackles non-convex matrix completion in semi-random settings.
problem Matrix completion in semi-random environments with varying observation probabilities.
method Proposes a pre-processing step to re-weight semi-random input, followed by a nearly-linear time algorithm.
result Recovering ground-truth matrix using non-convex local minima after pre-processing.
Paper tackles non-convex tensor regression with gradient descent.
problem Learning high-dimensional tensor regression with low-rank structure.
method Projected gradient descent on non-convex constraint set.
result Non-convex projected gradient descent provides superior statistical error and run-time.
Note on the computational complexity of Gromov-Wasserstein distance.
problem Computational difficulty of Gromov-Wasserstein distance.
method Analysis of the optimization problem structure and providing explicit examples.
result Gromov-Wasserstein distance optimization problem is non-convex quadratic.
A new non-convex method improves robust PCA with features.
problem Robust Principal Component Analysis with prior feature information.
method A novel non-convex optimization approach for decomposition.
result Exact recovery guarantees with low computational complexity.
SAGA achieves fast convergence on non-convex problems with RSC.
problem Non-convex optimization problems with restricted strong convexity.
method SAGA, a fast incremental gradient method, analyzed under RSC.
result SAGA achieves linear convergence up to statistical estimation accuracy under RSC.
This work shows neural networks can solve non-convex constraints problems.
problem Training neural networks under non-convex constraints.
method Project stochastic gradient descent with no-regret analysis of online learning.
result Overparameterized neural networks achieve near-optimal and near-feasible solutions.
Study accelerates optimization methods in non-convex problems, but doesn't improve the algorithm's performance.
problem Understanding the behavior of momentum-based acceleration methods in non-convex, high-dimensional landscapes.
method Used dynamical mean field theory to describe the average dynamics of heavy-ball momentum and Nesterov acceleration in a non-convex model.
result Accelerated dynamics but did not improve the algorithm's performance with respect to gradient descent.
Proposes TECU framework for efficient non-convex optimization.
problem Multivariate non-convex optimization problems with coupled objective functions.
method Embeds task-specific strategies into coordinate descent update schemes.
result Demonstrates improved efficiency and effectiveness in solving practical problems.
New framework shows all local minima are globally optimal in non-convex low-rank problems.
problem Non-convex low-rank problems, including matrix sensing, completion, and robust PCA.
method Developed a new framework to analyze the optimization landscapes of these problems.
result All local minima are also globally optimal and no high-order saddle points exist.
Here we study non-convex composite optimization: first, a finite-sum of smooth but non-convex functions, and second, a general function that admits a simple proximal mapping. Most research on stochastic methods for composite optimization assumes convexity or strong convexity of each function. In this paper, we extend t…
Paper proposes a working set algorithm for non-convex sparse regression with provable convergence.
problem Estimating sparse linear models from high-dimensional data using non-convex regularizers.
method FireWorks algorithm based on non-convex reformulation and leveraging residual geometry.
result Convergence to a stationary point of the full problem with provable guarantees.
New inexact proximal gradient methods solve non-convex optimization problems.
problem Solving non-convex optimization problems with non-smooth regularization.
method Proposed three inexact proximal gradient algorithms, including basic and Nesterov's accelerated versions.
result Theoretical analysis shows convergence rates similar to exact methods.
SVRG fails for deep learning due to non-convexity.
problem Applying SVRG to deep learning's non-convex optimization problems.
method Exploring SVRG and related techniques for deep learning.
result SVRG fails for deep learning's non-convex optimization problems.
Non-convex optimization problems often arise from probabilistic modeling, such as estimation of posterior distributions. Non-convexity makes the problems intractable, and poses various obstacles for us to design efficient algorithms. In this work, we attack non-convexity by first introducing the concept of \emph{probab…
Survey of advances in non-convex min-max optimization for applications.
problem Finding optimal solutions in non-convex, non-concave min-max problems.
method Selective review of theoretical and algorithmic advances.
result Exciting recent advances in solving non-convex min-max problems.
Novel framework shows exact recoverability of matrix completion and robust PCA with optimal sample complexity.
problem Efficiently recovering a hidden matrix from limited observations.
method Strong duality and novel analytical framework for non-convex matrix factorization problems.
result Exact recoverability and strong duality hold with nearly-optimal sample complexity guarantees for matrix completion and robust PCA.
A fast method for decentralized non-convex optimization over networks.
problem Decentralized non-convex optimization problems over a network of nodes.
method GT-SAGA, a randomized incremental gradient method that evaluates one component gradient per node per iteration.
result GT-SAGA achieves almost sure and mean-squared convergence to a first-order stationary point for general smooth non-convex problems.
Improved optimization guarantees for deep learning models with Nesterov acceleration.
problem Optimization in non-convex deep learning landscapes.
method Analysis of Nesterov acceleration in benignly non-convex landscapes.
result Identical guarantees can be obtained in optimization problems with weak geometric assumptions, especially in overparametrized deep learning.
Paper develops robust SGLD for solving non-convex DRO problems.
problem Solving non-convex distributionally robust optimisation problems with adversarially corrupted samples.
method Developed a Stochastic Gradient Langevin Dynamics (SGLD) algorithm with non-asymptotic convergence bounds.
result The robust SGLD estimator outperforms vanilla SGLD in terms of test accuracy.
Proves convergence of PSGLA for sampling non-convex potentials.
problem Sampling from non-convex potentials with stability.
method Combines ULA and proximal optimization with stability analysis.
result First proof of convergence for PSGLA on non-convex potentials.
Paper tackles non-convex optimization for higher moments in portfolio management.
problem Complexity of higher moments in optimization problems.
method Method of successive convex approximation.
result Solves mean-variance-skewness problem using non-convex optimization.
The TREX is a recently introduced method for performing sparse high-dimensional regression. Despite its statistical promise as an alternative to the lasso, square-root lasso, and scaled lasso, the TREX is computationally challenging in that it requires solving a non-convex optimization problem. This paper shows a remar…
We introduce a novel algorithm for solving learning problems where both the loss function and the regularizer are non-convex but belong to the class of difference of convex (DC) functions. Our contribution is a new general purpose proximal Newton algorithm that is able to deal with such a situation. The algorithm consi…
Study of Langevin processes and their convergence rates for non-convex problems.
problem Convergence of Langevin processes and SGD for non-convex optimization problems.
method Quantitative analysis of convergence rates for discrete Langevin-like processes.
result The convergence of SGD for non-convex problems depends on the potential function and additive noise.
Paper proposes efficient algorithm for non-convex rank minimization.
problem Efficiently solving rank minimization problems with non-convex penalties.
method Iterative Shrinkage-Thresholding Algorithm (ISTA) for non-convex weighted and reweighted nuclear norm.
result Proves convergence to critical point with rate O(1/T) and outperforms state-of-the-art methods. New method solves non-convex constrained optimization problems with non-differentiable constraints.
problem Training non-convex models with non-differentiable constraints.
method Proxy-Lagrangian formulation and semi-coarse correlated equilibrium.
result Solves non-convex constrained optimization problems with theoretical guarantees.
New diffusions help globally optimize non-convex functions.
problem Optimizing non-convex functions globally.
method Euler discretization of Langevin diffusion.
result Different diffusions optimize different convex and non-convex functions.
This work explores the non-convex optimization in compressive learning and the performance of heuristics.
problem The challenge of learning from compressed representations in compressive learning.
method Numerical simulations of the non-convex optimization landscape and heuristic performance.
result Properties of the non-convex optimization landscape and heuristic performance are explored.
Deep neural networks with multiple branches are less non-convex, improving performance.
problem Improving neural network performance through multi-branch architectures.
method Quantitative measurement of duality gap for neural networks with multi-branches and various activation functions.
result The duality gap of multi-branch neural networks decreases as the number of branches increases, leading to less non-convex optimization problems.
Gradient descent solves robust mean estimation in high dimensions.
problem High-dimensional robust mean estimation in the presence of adversarial outliers.
method Gradient descent with a structural lemma showing near-optimal solutions.
result Gradient descent can solve the robust mean estimation problem directly.
New method finds near-optimal solutions for non-convex optimization problems.
problem Finding near-optimal solutions for non-convex optimization problems.
method Riemannian stochastic recursive momentum method
result Achieves a near-optimal complexity of ildeO(ε−3). The paper develops variance-reduced methods to solve complex optimization problems.
problem Non-convex composition optimization with many inner functions.
method Variance-reduced techniques applied to SGD and SVRG.
result Significant improvement in query complexity for large inner function numbers.
AGGLIO optimizes non-convex functions with local convexity guarantees.
problem Optimizing non-convex functions with local convexity.
method Stage-wise, graduated optimization technique for locally convex functions.
result Global convergence to the global optimum for non-convex and locally convex objectives.