New algorithm tackles non-convex matrix completion in semi-random settings.
problem Matrix completion in semi-random environments with varying observation probabilities.
method Proposes a pre-processing step to re-weight semi-random input, followed by a nearly-linear time algorithm.
result Recovering ground-truth matrix using non-convex local minima after pre-processing.
SGD's uncertainty quantified in non-convex learning problems.
problem Uncertainty quantification in non-convex learning problems.
method Asymptotic normality of SGD iterates and bias characterization.
result SGD iterates are asymptotically normally distributed around the expected value of the invariant distribution.
New algorithm finds local minima in non-convex, non-smooth problems.
problem Finding local minimizers in non-convex and non-smooth optimization.
method Perturbed Proximal Descent, tailored for non-smooth cases.
result First known results for non-smooth optimization.
New algorithm improves convergence for non-convex problems with boundaries.
problem Optimizing non-convex problems with constraints.
method Reflected Gradient Langevin Dynamics with probabilistic representation.
result Promising convergence rates, faster than existing methods.
Paper tackles non-convex tensor regression with gradient descent.
problem Learning high-dimensional tensor regression with low-rank structure.
method Projected gradient descent on non-convex constraint set.
result Non-convex projected gradient descent provides superior statistical error and run-time.
New algorithms achieve high probability second-order convergence in non-convex optimization.
problem Stochastic non-convex optimization with high probability second-order convergence.
method Proposed NCG-S updating step and two algorithms.
result First algorithms with high probability second-order convergence and almost linear time complexity.
Non-convex sparsity-inducing penalties have recently received considerable attentions in sparse learning. Recent theoretical investigations have demonstrated their superiority over the convex counterparts in several sparse learning settings. However, solving the non-convex optimization problems associated with non-conv…
Paper proposes a working set algorithm for non-convex sparse regression with provable convergence.
problem Estimating sparse linear models from high-dimensional data using non-convex regularizers.
method FireWorks algorithm based on non-convex reformulation and leveraging residual geometry.
result Convergence to a stationary point of the full problem with provable guarantees.
This paper improves convergence guarantees for SGD algorithms in non-convex smooth functions.
problem Theoretical convergence properties of SGD algorithms for non-convex smooth functions.
method Analysis of SGD algorithms with arbitrary data ordering for non-convex smooth functions.
result Enhanced convergence guarantees for incremental gradient and single shuffle SGD, improving the optimization term of convergence guarantee.
This thesis explores how submodularity aids in optimizing non-convex functions and validating algorithms.
problem Understanding which functions can be optimized efficiently in non-convex settings.
method Introducing continuous submodularity and developing algorithms for maximizing these functions.
result Characterization and optimization of continuous submodular functions with strong guarantees.
Paper tackles non-convex inf-projection problems with stochastic optimization.
problem Non-convex and possibly non-smooth inf-projection minimization problems.
method Developed stochastic algorithms for finding (nearly) stationary solutions.
result Established first-order convergence for non-convex inf-projection problems.
New stochastic algorithms solve DC functions and non-convex problems efficiently.
problem Solving non-convex, non-smooth, and non-differentiable functions efficiently.
method Proposed new stochastic optimization algorithms for DC functions and non-convex problems.
result First non-asymptotic convergence for non-convex optimization with general non-convex non-differentiable regularizers.
Proves convergence of PSGLA for sampling non-convex potentials.
problem Sampling from non-convex potentials with stability.
method Combines ULA and proximal optimization with stability analysis.
result First proof of convergence for PSGLA on non-convex potentials.
New algorithm solves non-convex min-max problems in signal processing.
problem Non-convex min-max problems in signal processing and communication.
method Hybrid Block Successive Approximation (HiBSA) algorithm alternating gradient descent and ascent steps.
result HiBSA converges to first-order stationary solutions with global rates.
Asynchronous L-BFGS speeds up non-convex optimization.
problem Non-convex optimization challenges in machine learning.
method Asynchronous stochastic L-BFGS algorithm for non-convex optimization.
result Achieves an ergodic convergence rate of O ( 1 / N ) {\cal O}(1/\sqrt{N}) O ( 1/ N ) and linear speedup. This paper analyzes saddle points and minimax points in non-convex smooth games.
problem Understanding local optimal points in non-convex smooth games.
method Comprehensive analysis of local minimax points, including their optimality conditions and stability.
result Local saddle points are uniformly local minimax points under mild continuity assumptions.
New inexact proximal gradient methods solve non-convex optimization problems.
problem Solving non-convex optimization problems with non-smooth regularization.
method Proposed three inexact proximal gradient algorithms, including basic and Nesterov's accelerated versions.
result Theoretical analysis shows convergence rates similar to exact methods.
Flexible ADMM-based algorithm for non-convex problems with convergence guarantees.
problem Solving non-convex problems with smooth and convex components.
method Incorporates ADMM, SCA, distributed, asynchronous, and inexact gradient methods.
result Establishes first-order convergence rate guarantees under mild assumptions.
A new non-convex method improves robust PCA with features.
problem Robust Principal Component Analysis with prior feature information.
method A novel non-convex optimization approach for decomposition.
result Exact recovery guarantees with low computational complexity.
Faster algorithm finds approximate local minima.
problem Finding approximate local minima in non-convex optimization problems.
method Non-convex second-order optimization algorithm.
result Finds approximate local minima faster than gradient descent.
FTPL achieves optimal regret in online non-convex learning.
problem Online non-convex learning with non-convex losses.
method Follow the Perturbed Leader (FTPL) algorithm.
result FTPL achieves optimal regret rate of O ( T − 1 / 2 ) O(T^{-1/2}) O ( T − 1/2 ) . We introduce a novel algorithm for solving learning problems where both the loss function and the regularizer are non-convex but belong to the class of difference of convex (DC) functions. Our contribution is a new general purpose proximal Newton algorithm that is able to deal with such a situation. The algorithm consi…
New algorithms optimize non-smooth, non-convex objectives with improved complexity.
problem Optimizing non-smooth, non-convex stochastic objectives.
method Reduction to online learning, applying optimistic online learning techniques.
result Improved complexity for finding ( δ , ε ) (δ,ε) ( δ , ε ) -stationary points. New algorithm TUSLA improves learning of non-convex neural networks.
problem Optimizing non-convex loss functions in neural networks with superlinear gradient growth.
method Tamed Unadjusted Stochastic Langevin Algorithm (TUSLA) based on SGLD with taming technology.
result Finite-time guarantees for TUSLA to find approximate minimizers of empirical and population risks.
Proposes TECU framework for efficient non-convex optimization.
problem Multivariate non-convex optimization problems with coupled objective functions.
method Embeds task-specific strategies into coordinate descent update schemes.
result Demonstrates improved efficiency and effectiveness in solving practical problems.
Paper proposes efficient algorithm for non-convex rank minimization.
problem Efficiently solving rank minimization problems with non-convex penalties.
method Iterative Shrinkage-Thresholding Algorithm (ISTA) for non-convex weighted and reweighted nuclear norm.
result Proves convergence to critical point with rate O ( 1 / T ) O(1/T) O ( 1/ T ) and outperforms state-of-the-art methods. New algorithm STORM reduces variance in non-convex optimization without large batches.
problem Improving convergence in non-convex optimization problems.
method Adaptive learning rates and momentum-based variance reduction.
result Achieves optimal convergence rate without batch sizes or knowledge of variance.
The paper analyzes adaptive algorithms in non-convex optimization landscapes.
problem Analyzing adaptive algorithms in non-convex optimization landscapes.
method Stochastic algorithms with decreasing step-size, considering mini-batches and noise.
result Established almost sure convergence to critical points and minimizers.
Study accelerates optimization methods in non-convex problems, but doesn't improve the algorithm's performance.
problem Understanding the behavior of momentum-based acceleration methods in non-convex, high-dimensional landscapes.
method Used dynamical mean field theory to describe the average dynamics of heavy-ball momentum and Nesterov acceleration in a non-convex model.
result Accelerated dynamics but did not improve the algorithm's performance with respect to gradient descent.
New approach for distributed online optimization of non-convex losses with sublinear regret.
problem Regret evaluation and consensus in distributed, multi-agent systems with non-convex losses.
method Composite regret metric and consensus-based online normalized gradient (CONGD) approach for pseudo-convex losses; offline optimization oracle for general non-convex losses.
result First sublinear regret bound for general distributed online non-convex learning.
Paper develops momentum schemes with variance reduction for non-convex composition optimization.
problem Lack of convergence guarantee and efficient momentum design in existing algorithms.
method Develops various momentum schemes with SPIDER-based variance reduction.
result Achieves near-optimal sample complexity and linear convergence rate.
Paper tackles non-convex constrained DRO with a stochastic algorithm for large-scale applications.
problem Training robust models against data distribution shifts with non-convex loss functions.
method Developed a stochastic algorithm for non-convex constrained DRO with a complexity independent of dataset size.
result Algorithm finds ε-stationary points with computational complexity of O(ε^(-3k_*-5)) for general Cressie-Read divergence.
First order methods can take extremely long to find global minima of non-convex functions.
problem Finding global minimizers of non-convex functions.
method Designing a family of non-convex functions and using statistical lower bounds for parameter estimation.
result First order methods can take exponential time to converge to a global minimizer.
Improved DP algorithms for non-convex optimization with tighter generalization bounds.
problem Private stochastic non-convex optimization in high-dimensional spaces.
method Differential privacy techniques, including adaptive algorithms like DP RMSProp and DP Adam, combined with adaptive data analysis.
result Achieved a sharper rate of p 4 / n \sqrt[4]{p}/\sqrt{n} 4 p / n for population loss, improving upon previous bounds. Improved optimization guarantees for deep learning models with Nesterov acceleration.
problem Optimization in non-convex deep learning landscapes.
method Analysis of Nesterov acceleration in benignly non-convex landscapes.
result Identical guarantees can be obtained in optimization problems with weak geometric assumptions, especially in overparametrized deep learning.
Adaptive momentum method solves non-convex min-max problems.
problem Non-convex min-max optimization problems in training generative adversarial networks.
method Proposes an adaptive momentum algorithm for non-convex min-max optimization.
result Establishes non-asymptotic convergence rates for the proposed algorithm.
This study improves graph signal denoising for vector-valued data with non-convex penalties.
problem Denoising piecewise smooth graph signals with varying smoothness levels.
method Extended graph trend filtering with non-convex penalties and ADMM algorithm.
result Non-convex penalties outperform convex ones in recovery performance.
This monograph explores non-convex optimization techniques for machine learning.
problem Capturing complex learning and prediction problems with non-convex optimization.
method Analyzes various non-convex optimization techniques and their applications.
result Direct non-convex optimization methods often outperform relaxation-based techniques.
New algorithm SFHC achieves near-optimal costs with predictions for non-convex optimization.
problem Online optimization with non-convex hitting costs and movement costs.
method Synchronized Fixed Horizon Control (SFHC) algorithm with conditions on hitting and movement costs.
result Synchronized Fixed Horizon Control (SFHC) achieves a 1 + O ( 1 / w ) 1+O(1/w) 1 + O ( 1/ w ) competitive ratio for near-optimal costs. Improved SdLBFGS converges better and stabilizes optimization in PyTorch.
problem Stochastic non-convex optimization in PyTorch.
method Modified SdLBFGS with identity matrix initialization and direction normalization.
result Improved SdLBFGS converges better and is more stable.
Reduces non-convex optimization to finding local minima using gradients.
problem Finding local minima in non-convex optimization problems.
method Reduces non-convex optimization to gradient-based methods.
result Turns various optimization algorithms into local minimum finding methods.
Paper develops robust SGLD for solving non-convex DRO problems.
problem Solving non-convex distributionally robust optimisation problems with adversarially corrupted samples.
method Developed a Stochastic Gradient Langevin Dynamics (SGLD) algorithm with non-asymptotic convergence bounds.
result The robust SGLD estimator outperforms vanilla SGLD in terms of test accuracy.
New algorithms solve non-convex isotonic regression problems efficiently.
problem Minimizing submodular functions with ordering constraints.
method Discretization schemes leading to zero-th, first, or higher order oracles for efficient optimization.
result Non-convex loss functions can be robust to outliers and still lead to efficient optimization.
In this paper we develop proximal methods for statistical learning. Proximal point algorithms are useful in statistics and machine learning for obtaining optimization solutions for composite functions. Our approach exploits closed-form solutions of proximal operators and envelope representations based on the Moreau, Fo…
This paper examines the role and efficiency of the non-convex loss functions for binary classification problems. In particular, we investigate how to design a simple and effective boosting algorithm that is robust to the outliers in the data. The analysis of the role of a particular non-convex loss for prediction accur…
Non-convex optimization problems often arise from probabilistic modeling, such as estimation of posterior distributions. Non-convexity makes the problems intractable, and poses various obstacles for us to design efficient algorithms. In this work, we attack non-convexity by first introducing the concept of \emph{probab…
New algorithm learns optimal stepsizes for SGD in noisy non-convex optimization.
problem Finding optimal stepsize for SGD in noisy non-convex optimization.
method Surrogate losses cast problem into online convex optimization, using no-regret algorithms.
result Self-tuned SGD algorithm with adaptive convergence rates.
Dual IHT algorithm solves NP-hard non-convex sparse minimization problems.
problem Non-convex sparse minimization with ℓ 2 \ell_2 ℓ 2 -regularized loss function. method Developed a dual IHT algorithm for maximizing the non-smooth dual objective.
result Sparse recovery performance is invariant to RIP, superior to primal IHT algorithms.