New method simplifies optimization landscapes by transforming saddle points.
problem Saddle points hinder non-convex optimization in machine learning.
method Variable elimination algorithms, like VarPro, are compared to reveal geometric insights.
result Variable elimination reshapes critical point structure, creating local maxima from saddle points.
The paper solves utility maximization under partial information using transformations and perturbation methods.
problem Maximizing recursive utility under partial information.
method Transforming to full information, using variational formulation, stochastic game approach, and terminal perturbation method.
result Explicit saddle points and optimal terminal wealth obtained.
Paper analyzes Transformer learning dynamics, proving benign landscape for in-context learning.
problem Understanding how Transformers learn in context with nonlinear features.
method Mean-field and two-timescale analysis of Transformer dynamics, proving nonconvex but benign landscape.
result Proves mean-field dynamics avoid saddle points, leading to improved optimization.
DEO uses gradient information to escape saddle points in neural networks.
problem Training deep neural networks struggles with flat regions and saddle points.
method Dimer-Enhanced Optimization (DEO) uses gradient information to estimate curvature and escape saddle points.
result DEO improves training efficiency and performance compared to standard first-order methods.
Movie moves connect singular link cobordisms in 4D space.
problem Tackling the concept of singular link cobordisms in 4D space.
method Presenting a set of movie moves to connect any two movies of isotopic singular link cobordisms.
result A set of movie moves sufficient to connect any two movies of isotopic singular link cobordisms.
New saddle network architectures preserve convex-concave geometry in optimization problems.
problem Optimization models with convex x and concave y components.
method Structured separable decomposition and saddle network architectures.
result Proven one-dimensional approximation theorem and high accuracy on various test functions.
The paper optimizes wealth with concave coefficients in a recursive utility maximization problem.
problem Optimizing wealth with concave coefficients in a recursive utility maximization problem.
method Equivalent backward formulation, Fenchel-Legendre transform, convex duality method.
result Derives the optimal terminal wealth for investors with ambiguity aversion.
The paper presents methods to improve policy evaluation in reinforcement learning.
problem Policy evaluation in reinforcement learning with linear function approximation.
method Transformed into a convex-concave saddle point problem, primal-dual batch gradient method, and two stochastic variance reduction methods.
result Achieved linear convergence even with only strong concavity in dual variables.
Paper analyzes algorithms for nonstationary saddle-point optimization problems.
problem Nonstationary saddle-point optimization problems in game theory, reinforcement learning, and machine learning.
method Proposes extragradient and Frank-Wolfe algorithms for online and bandit settings.
result Establishes sub-linear regret bounds for the proposed algorithms.
Gradient-based methods struggle with saddle points; curvature exploitation helps.
problem Gradient-based methods struggle with saddle points, leading to undesired stable stationary points.
method Exploits curvature information to escape undesired stationary points.
result Different optimization methods, including gradient and Adagrad, can escape non-optimal stationary points when curvature exploitation is used.
Heavy-ball algorithms can always avoid saddle points with random initialization.
problem Optimizing nonconvex functions with saddle points.
method Developed a new mapping to interpret heavy-ball algorithms as iterations, proving they can escape saddle points.
result Heavy-ball algorithms can escape saddle points with random initialization.
Develops algorithm to escape saddle points in Byzantine settings.
problem Byzantine workers create fake local minima near saddle points.
method ByzantinePGD, a robust first-order algorithm.
result Converges to approximate true local minimizer with low iteration complexity.
FeDualEx tackles saddle point optimization in federated learning with composite objectives.
problem Saddle point optimization with constraints and non-smooth regularization in federated learning.
method Federated Dual Extrapolation (FeDualEx) algorithm for saddle point optimization and composite objectives.
result FeDualEx effectively solves saddle point optimization problems with composite objectives in federated learning.
Gradient descent can be very slow in escaping saddle points.
problem Gradient descent's slowness in escaping saddle points.
method Gradient descent with natural random initialization and non-pathological functions.
result Gradient descent can take exponential time to escape saddle points.
Unified analysis of EG and OGDA for saddle point problems using proximal point method.
problem Solving saddle point problems in bilinear and strongly convex-strongly concave settings.
method Unified analysis as approximations of the proximal point method.
result Unified analysis of EG and OGDA for saddle point problems.
First-order methods avoid saddle points for most initializations.
problem Avoiding saddle points in optimization problems.
method First-order methods, including gradient descent and variants, analyzed using dynamical systems and the Stable Manifold Theorem.
result First-order methods avoid saddle points for almost all initializations.
Paper proposes efficient method to escape saddle points in non-convex optimization.
problem Hardness of escaping saddle points in non-convex optimization.
method Designs an efficient algorithm using higher order derivatives to converge to third order local optima.
result First efficient algorithm guaranteed to converge to a third order local optimum.
New algorithm tackles saddle points in deep learning.
problem Saddle points proliferation in nonconvex optimization.
method Hessian-free optimization, addressing computational complexity and saddle point issue.
result First step towards applying Newton's method to nonconvex optimization.
A new method helps escape saddle points in non-convex optimization.
problem Escaping saddle points in non-convex optimization problems.
method CNC-SCSG method using a separate SGD step to help escape from strict saddle points.
result The method converges to a second-order stationary point with a rate of O(ε−2log(1/ε)). Gradient descent can take exponentially long to escape saddle points in 2D.
problem Worst-case inefficiency of gradient descent in non-convex optimization.
method Analysis of gradient descent's performance on 2D functions.
result Gradient descent can take exponentially long to escape saddle points.
A new method avoids saddle points in training machine learning models.
problem Training machine learning models efficiently in the presence of saddle points.
method Modified Laplacian smoothing gradient descent (mLSGD).
result The attraction region for mLSGD is significantly smaller than for gradient descent, avoiding saddle points.
Perturbed gradient descent escapes saddle points efficiently.
problem Escaping saddle points in optimization problems.
method Perturbed gradient descent approach.
result Escaping saddle points almost for free, matching convergence rate of gradient descent.
New methods help escape strict saddle points in nonsmooth optimization.
problem Escaping strict saddle points in nonsmooth optimization.
method An inexact stochastically perturbed gradient method applied to the Moreau envelope.
result A variety of algorithms for nonsmooth optimization can efficiently escape strict saddle points of the Moreau envelope.
New insights into matrix factorization show strict saddles have bounded eigenvalues.
problem Understanding the nature of critical points in matrix factorization.
method Analyzing orbits of critical points under the general linear group and identifying canonical points.
result Minimum eigenvalue of strict saddles is not uniformly bounded below zero.
New algorithm speeds up solving saddle-point problems with large condition numbers.
problem Solving saddle-point problems with large condition numbers.
method Proposes a stochastic proximal point algorithm that accelerates variance reduction methods.
result Reduces logarithmic term of condition number for iteration complexity.
A new algorithm trains deep neural networks by adding neurons greedily.
problem Training deep neural networks efficiently and effectively.
method Neuron Pursuit (NP) algorithm, which alternates between neuron addition and loss minimization.
result The algorithm can train deep neural networks efficiently and effectively.
New method solves saddle-point problems faster than existing methods.
problem Large-scale saddle-point problems in optimization.
method Sequential subspace optimization with proximal regularization.
result Significantly better convergence compared to first-order methods.
Improved Frank-Wolfe algorithm solves saddle point problems efficiently.
problem Solving constrained smooth convex-concave saddle point problems.
method Extends Frank-Wolfe algorithm to use linear minimization oracles.
result First proof of convergence for FW-type saddle point solver over polytopes.
This paper extends Newton's method to distributed learning, avoiding saddle points and handling Byzantine workers.
problem Avoiding saddle points in distributed non-convex optimization, especially in the presence of Byzantine workers.
method Extends cubic-regularized Newton method to distributed framework, addressing communication bottlenecks and Byzantine attacks.
result The method achieves improved iteration complexity compared to first-order methods, with a 25% improvement in experiments.
PWGF escapes saddle points in nonconvex optimization.
problem Escaping saddle points in nonconvex optimization.
method PWGF uses noisy perturbations via Gaussian process to escape saddle points.
result PWGF achieves second-order optimality for nonconvex objectives.
DLNs dynamics change with variance, leading to saddle-to-saddle training phases.
problem Understanding the dynamics of DLNs with varying initialization variance.
method Analyzing the phase transition of DLNs' dynamics as variance changes.
result Gradient descent visits a sequence of saddles, reaching a sparse global minimum.
This paper develops methods to solve saddle-point problems on Riemannian manifolds with exponential stability.
problem Solving saddle-point problems on Riemannian manifolds with exponential stability.
method Developed a projected dynamical system on a Riemannian manifold to solve saddle-point problems, leveraging the strong monotonicity of the gradient of the Lagrangian function.
result Established exponential stability and convergence of the projected dynamical system to the unique saddle-point.
Last iterate of Extragradient algorithm converges slower than averaged iterates in saddle point problems.
problem Smooth convex-concave saddle point problems
method Analysis of Extragradient (EG) algorithm convergence rates
result The last iterate of EG converges at a rate of O(1/√T), compared to O(1/T) for averaged iterates
New ODE models show saddle-point optimization methods converge differently, with last-iterate convergence for OGDA.
problem Analyzing convergence properties of saddle-point optimization methods.
method High-Resolution Differential Equations (HRDEs) to design differential equation models for saddle-point optimization methods.
result HRDEs reveal last-iterate convergence for Optimistic Gradient Descent Ascent (OGDA) in bilinear games.
We optimize saddle-point problems for large-scale Markov decision processes.
problem Optimizing policies in large-scale Markov decision processes.
method Characterized conditions for convergence and designed an optimization algorithm.
result Our algorithm converges faster and is state-space independent.
NGD avoids saddle points, leading to faster convergence.
problem Optimizing non-convex functions to avoid saddle points.
method Normalized Gradient Descent (NGD) with noise injection.
result NGD provably evades saddle points and converges to local minima.
Study on neural networks in overparameterized cases, focusing on flat minima and saddle points.
problem Understanding the landscape of training error in neural networks with overparameterization.
method Three methods of embedding a network into a wider one with more hidden units, analyzing the embedded point's properties.
result Smooth and ReLU activation networks have different partially flat landscapes around the embedded point.
Houdini finds high-dimensional saddle points under few constraints.
problem Escaping from saddle points in high-dimensional spaces with constraints.
method Gradient descent methods under logarithmic inequality constraints.
result Polynomial time algorithms for escaping saddle points under constraints.
Extends saddle-point method for large-time volatility smiles.
problem Analyzing large-time volatility smiles in financial models.
method Saddle-point approach to derive large-time model-implied volatility smiles.
result Provides theoretical foundation and wide class of arbitrage-free parametrizations.
Gradient descent can use larger step sizes to avoid strict saddle points.
problem Avoiding strict saddle points in non-convex optimization.
method Proving that gradient descent with step-size up to 2/L avoids strict saddle points with high probability.
result Gradient descent with step-size up to 2/L almost surely avoids strict saddle points.
Deep ReLU networks escape from the origin via saddle points with a low-rank bias.
problem Understanding the dynamics of gradient descent in deep ReLU networks.
method Analysis of escape directions and singular values of weight matrices.
result The first singular value of the ℓ-th layer weight matrix is at least ℓ41 larger than any other singular value. Efficient method solves large-scale saddle point problems with parallel updates.
problem Large-scale convex-concave saddle point problems with separable structure.
method Stochastic parallel block coordinate descent with adaptive primal-dual updates.
result Significantly better performance than state-of-the-art methods in various applications.
Paper defines saddle points in asymmetric Dynkin games using martingale theory.
problem Tackles saddle point conditions in asymmetric Dynkin games with partial information.
method Uses martingale theory to identify super and submartingales related to equilibrium payoffs.
result Characterizes saddle point strategies in terms of equilibrium payoffs' dynamics and Doob-Meyer decompositions.
New algorithm solves saddle point problems in Banach spaces.
problem Solving saddle point problems in real reflexive Banach spaces.
method Stochastic Bregman Primal-Dual Splitting Algorithm with relative smoothness and strong convexity assumptions.
result Almost sure convergence to saddle points under various conditions.
Algorithm classifies saddle-focus singularities in Hamiltonian systems.
problem Classifying nondegenerate saddle-focus singularities in integrable Hamiltonian systems.
method Developed an algorithm based on semi-local equivalence to represent singularities as almost direct products.
result Obtained complete lists of saddle-focus singularities of complexities 1, 2, and 3.
Adaptive methods like Adam help escape saddle points in deep learning.
problem Escaping saddle points in nonconvex optimization problems.
method Viewed as preconditioned SGD, where the preconditioner estimates noise isotropy.
result Adaptive methods can efficiently estimate a preconditioner that helps escape saddle points.
Classifies Morse flows on 3-sphere with specific saddle connections.
problem Classifying Morse-Smale flows on a 3-sphere with specific saddle connections.
method Used generalized Heegaard diagrams (Pr-diagrams) to classify flows.
result Found all possible, up to homeomorphism, ways to embed two circles in a 2-sphere with no more than 10 points of transversal intersection.
The paper studies neural networks' convergence near origin and saddle points.
problem Directional convergence of neural networks near small initializations and saddle points.
method Gradient flow dynamics analysis of two-homogeneous neural networks.
result Neural networks' weights approximately converge in direction to KKT points for small initializations.