Heavy-ball algorithms can always avoid saddle points with random initialization.
problem Optimizing nonconvex functions with saddle points.
method Developed a new mapping to interpret heavy-ball algorithms as iterations, proving they can escape saddle points.
result Heavy-ball algorithms can escape saddle points with random initialization.
Paper analyzes algorithms for nonstationary saddle-point optimization problems.
problem Nonstationary saddle-point optimization problems in game theory, reinforcement learning, and machine learning.
method Proposes extragradient and Frank-Wolfe algorithms for online and bandit settings.
result Establishes sub-linear regret bounds for the proposed algorithms.
Develops algorithm to escape saddle points in Byzantine settings.
problem Byzantine workers create fake local minima near saddle points.
method ByzantinePGD, a robust first-order algorithm.
result Converges to approximate true local minimizer with low iteration complexity.
New algorithm speeds up solving saddle-point problems with large condition numbers.
problem Solving saddle-point problems with large condition numbers.
method Proposes a stochastic proximal point algorithm that accelerates variance reduction methods.
result Reduces logarithmic term of condition number for iteration complexity.
Improved Frank-Wolfe algorithm solves saddle point problems efficiently.
problem Solving constrained smooth convex-concave saddle point problems.
method Extends Frank-Wolfe algorithm to use linear minimization oracles.
result First proof of convergence for FW-type saddle point solver over polytopes.
FeDualEx tackles saddle point optimization in federated learning with composite objectives.
problem Saddle point optimization with constraints and non-smooth regularization in federated learning.
method Federated Dual Extrapolation (FeDualEx) algorithm for saddle point optimization and composite objectives.
result FeDualEx effectively solves saddle point optimization problems with composite objectives in federated learning.
Unified analysis of EG and OGDA for saddle point problems using proximal point method.
problem Solving saddle point problems in bilinear and strongly convex-strongly concave settings.
method Unified analysis as approximations of the proximal point method.
result Unified analysis of EG and OGDA for saddle point problems.
Paper proposes efficient method to escape saddle points in non-convex optimization.
problem Hardness of escaping saddle points in non-convex optimization.
method Designs an efficient algorithm using higher order derivatives to converge to third order local optima.
result First efficient algorithm guaranteed to converge to a third order local optimum.
New methods help escape strict saddle points in nonsmooth optimization.
problem Escaping strict saddle points in nonsmooth optimization.
method An inexact stochastically perturbed gradient method applied to the Moreau envelope.
result A variety of algorithms for nonsmooth optimization can efficiently escape strict saddle points of the Moreau envelope.
A new algorithm trains deep neural networks by adding neurons greedily.
problem Training deep neural networks efficiently and effectively.
method Neuron Pursuit (NP) algorithm, which alternates between neuron addition and loss minimization.
result The algorithm can train deep neural networks efficiently and effectively.
Last iterate of Extragradient algorithm converges slower than averaged iterates in saddle point problems.
problem Smooth convex-concave saddle point problems
method Analysis of Extragradient (EG) algorithm convergence rates
result The last iterate of EG converges at a rate of O(1/√T), compared to O(1/T) for averaged iterates
GenFlow optimizes faster, avoiding saddle points in fixed time.
problem Designing efficient optimization algorithms for convex and non-convex functions.
method Introduces GenFlow and momentum variants with fixed-time convergence guarantees.
result GenFlow and momentum variants converge to optimal solutions in fixed time for PL functions and evade saddle points uniformly.
Algorithm classifies saddle-focus singularities in Hamiltonian systems.
problem Classifying nondegenerate saddle-focus singularities in integrable Hamiltonian systems.
method Developed an algorithm based on semi-local equivalence to represent singularities as almost direct products.
result Obtained complete lists of saddle-focus singularities of complexities 1, 2, and 3.
New algorithm solves saddle point problems in Banach spaces.
problem Solving saddle point problems in real reflexive Banach spaces.
method Stochastic Bregman Primal-Dual Splitting Algorithm with relative smoothness and strong convexity assumptions.
result Almost sure convergence to saddle points under various conditions.
This paper analyzes saddle points and minimax points in non-convex smooth games.
problem Understanding local optimal points in non-convex smooth games.
method Comprehensive analysis of local minimax points, including their optimality conditions and stability.
result Local saddle points are uniformly local minimax points under mild continuity assumptions.
Simple gradient descent algorithm escapes saddle points efficiently.
problem Escaping saddle points in nonconvex optimization.
method Gradient-based algorithm with polynomial iterations.
result Outputs ε-approximate second-order stationary points efficiently.
A new method helps escape saddle points in non-convex optimization.
problem Escaping saddle points in non-convex optimization problems.
method CNC-SCSG method using a separate SGD step to help escape from strict saddle points.
result The method converges to a second-order stationary point with a rate of O(ε−2log(1/ε)). First-order methods avoid saddle points for most initializations.
problem Avoiding saddle points in optimization problems.
method First-order methods, including gradient descent and variants, analyzed using dynamical systems and the Stable Manifold Theorem.
result First-order methods avoid saddle points for almost all initializations.
New algorithm finds local minima in non-convex, non-smooth problems.
problem Finding local minimizers in non-convex and non-smooth optimization.
method Perturbed Proximal Descent, tailored for non-smooth cases.
result First known results for non-smooth optimization.
New algorithm helps escape saddle points in optimization problems.
problem Optimizing smooth non-convex functions to avoid saddle points.
method Perturbed Saddle-escape Descent (PSD) algorithm with explicit constants.
result PSD finds approximate second-order stationary points efficiently.
Gradient descent can take exponentially long to escape saddle points in 2D.
problem Worst-case inefficiency of gradient descent in non-convex optimization.
method Analysis of gradient descent's performance on 2D functions.
result Gradient descent can take exponentially long to escape saddle points.
New method solves saddle-point problems faster than existing methods.
problem Large-scale saddle-point problems in optimization.
method Sequential subspace optimization with proximal regularization.
result Significantly better convergence compared to first-order methods.
New ASGLD algorithm escapes saddle points and converges to local minima.
problem Non-convex optimization problems with saddle points.
method Adaptive stochastic gradient Langevin dynamics (ASGLD) and its specialized versions ASG and AGLD.
result ASGLD and ASG escape saddle points in nearly dimension-free iterations and converge to local minima.
New method avoids saddle points without gradients.
problem Optimizing non-convex functions efficiently.
method Zero-order derivative-free algorithm using only function evaluations.
result Converges to second-order stationary points efficiently.
Paper tackles zeroth-order optimization for nonconvex problems with constraints, high-dimensions, and saddle-points.
problem Optimization of nonconvex functions with constraints and high-dimensionality, avoiding saddle-points.
method Proposes zeroth-order stochastic approximation algorithms, including conditional gradient and truncated gradient methods, and a zeroth-order cubic regularization Newton's method.
result Demonstrates algorithms achieving rates similar to standard stochastic gradient methods, with rates dependent on poly-logarithmic dimensionality.
New method simplifies optimization landscapes by transforming saddle points.
problem Saddle points hinder non-convex optimization in machine learning.
method Variable elimination algorithms, like VarPro, are compared to reveal geometric insights.
result Variable elimination reshapes critical point structure, creating local maxima from saddle points.
Houdini finds high-dimensional saddle points under few constraints.
problem Escaping from saddle points in high-dimensional spaces with constraints.
method Gradient descent methods under logarithmic inequality constraints.
result Polynomial time algorithms for escaping saddle points under constraints.
NGD avoids saddle points, leading to faster convergence.
problem Optimizing non-convex functions to avoid saddle points.
method Normalized Gradient Descent (NGD) with noise injection.
result NGD provably evades saddle points and converges to local minima.
Algorithm minimizes SP-Regret for online saddle point problem and related knapsack optimization.
problem Online saddle point problem and related online convex optimization with knapsacks.
method Proposed algorithms achieving sublinear SP-Regret in various settings.
result Achieved sublinear SP-Regret bounds for different problem settings.
Optimal privacy-preserving algorithm for solving saddle point problems.
problem Solving convex-concave stochastic saddle point problems under differential privacy constraints.
method Recursive regularization technique repurposed for saddle point problems, achieving strong gap rate of O(1/√n + √d/nε).
result Achieves nearly optimal strong gap rate of O(1/√n + √d/nε) with gradient complexity O(min{n^2ε^(1.5)/√d, n^(3/2)}).
We optimize saddle-point problems for large-scale Markov decision processes.
problem Optimizing policies in large-scale Markov decision processes.
method Characterized conditions for convergence and designed an optimization algorithm.
result Our algorithm converges faster and is state-space independent.
PWGF escapes saddle points in nonconvex optimization.
problem Escaping saddle points in nonconvex optimization.
method PWGF uses noisy perturbations via Gaussian process to escape saddle points.
result PWGF achieves second-order optimality for nonconvex objectives.
Gradient descent can use larger step sizes to avoid strict saddle points.
problem Avoiding strict saddle points in non-convex optimization.
method Proving that gradient descent with step-size up to 2/L avoids strict saddle points with high probability.
result Gradient descent with step-size up to 2/L almost surely avoids strict saddle points.
New analysis shows GD and SGD avoid saddle points efficiently in high dimensions.
problem Gradient descent and SGD struggle with saddle points in high-dimensional nonconvex optimization problems.
method Perturbed versions of GD and SGD analyzed for efficiency in high dimensions.
result Perturbed GD and SGD converge to second-order stationary points efficiently, avoiding saddle points.
This paper extends Newton's method to distributed learning, avoiding saddle points and handling Byzantine workers.
problem Avoiding saddle points in distributed non-convex optimization, especially in the presence of Byzantine workers.
method Extends cubic-regularized Newton method to distributed framework, addressing communication bottlenecks and Byzantine attacks.
result The method achieves improved iteration complexity compared to first-order methods, with a 25% improvement in experiments.
New theory explains how momentum SGD helps avoid saddle points in nonconvex optimization.
problem Understanding convergence properties of Momentum SGD in nonconvex optimization.
method Diffusion approximations for nonconvex optimization problems with strict saddle points and isolated local optima.
result Momentum helps escape from saddle points but hurts convergence near optima.
The paper shows tree models are vulnerable to adversarial examples and develops a robust algorithm.
problem Vulnerability of tree-based models to adversarial examples.
method Develops a novel algorithm to learn robust trees by optimizing performance under worst-case perturbation of input features.
result The proposed algorithms substantially improve the robustness of tree-based models against adversarial examples.
Agents learn shared dictionary elements and parameters in a decentralized online setting.
problem Discriminative dictionary learning in a distributed online setting.
method Formulated as a distributed stochastic program, solved using a block variant of the Arrow-Hurwicz saddle point algorithm with Lagrange multipliers.
result Decisions asymptotically achieve a first-order stationarity condition on average.
Nonconvex optimization problems such as the ones in training deep neural networks suffer from a phenomenon called saddle point proliferation. This means that there are a vast number of high error saddle points present in the loss function. Second order methods have been tremendously successful and widely adopted in the…
New first-order algorithm escapes saddle points faster than existing methods.
problem Escaping from saddle points in optimization problems.
method Integrates noise into first-order information to extract negative curvature from Hessian.
result First-order stochastic algorithm achieves almost linear time complexity for finding near second-order stationary points.
AGD outperforms GD in escaping saddle points faster.
problem Finding second-order stationary points in nonconvex optimization.
method A simple variant of Nesterov's accelerated gradient descent (AGD).
result AGD finds a second-order stationary point in ildeO(1/ε7/4) iterations, faster than GD's ildeO(1/ε2). Riemannian gradient descent helps escape saddle points on curved spaces.
problem Minimizing nonconvex functions on curved spaces (Riemannian manifolds).
method Perturbed Riemannian gradient descent algorithm.
result Converges to second-order stationary points, matching unconstrained smooth minimization rates.
Gradient-based methods struggle with saddle points; curvature exploitation helps.
problem Gradient-based methods struggle with saddle points, leading to undesired stable stationary points.
method Exploits curvature information to escape undesired stationary points.
result Different optimization methods, including gradient and Adagrad, can escape non-optimal stationary points when curvature exploitation is used.
New method stabilizes saddle-point optimization with unbounded gradients.
problem Stochastic saddle-point optimization faces instability due to large gradients.
method Proposes a regularization technique to stabilize iterates.
result Yields meaningful performance guarantees even with unbounded gradients.
Decentralized method solves saddle point problems with theoretical guarantees.
problem Solving saddle point problems in a decentralized network.
method Proximal point method adapted for decentralized networks.
result Converges to approximate stationarity with rate of O(1/√T).
The paper analyzes GTD algorithms with finite-sample bounds.
problem Convergence rate analysis of GTD family of algorithms.
method Formulated as stochastic gradient algorithms and analyzed using saddle-point error.
result Obtained finite-sample bounds on GTD performance.
Two new algorithms improve reinforcement learning by avoiding saddle points.
problem Control in reinforcement learning without model information.
method Cubic-regularized Policy Newton algorithms using likelihood ratio method.
result Convergence to a second-order stationary point with improved sample complexity.
Oracle-efficient algorithm for offline RL with partial data coverage.
problem Offline reinforcement learning with partial data coverage and constraints.
method PDOCRL, a primal-dual algorithm with decomposed linear-programming formulation.
result Near-optimal, near-feasible policy with \(\widetilde{\mathcal O}(ε^{-2})\) sample guarantee.