arXiv research

Study proves existence and properties of shrinkers in area-preserving curve-shortening flow.

New ODE models show saddle-point optimization methods converge differently, with last-iterate convergence for OGDA.

Equivalence of convex optimization, saddle-point problems, and variational inequalities is a well-established concept. The variational inequality (VI) is a static problem which is studied under dynamical settings using a framework called the projected dynamical system, whose stationary points coincide with the static s…

GenFlow optimizes faster, avoiding saddle points in fixed time.

We analyze stochastic gradient descent for optimizing non-convex functions. In many cases for non-convex functions the goal is to find a reasonable local minimum, and the main concern is that gradient updates are trapped in saddle points. In this paper we identify strict saddle property for non-convex problem that allo…

In this paper we study the smooth convex-concave saddle point problem. Specifically, we analyze the last iterate convergence properties of the Extragradient (EG) algorithm. It is well known that the ergodic (averaged) iterates of EG converge at a rate of $O(1/T)$ (Nemirovski, 2004). In this paper, we show that the last…

We examine the squared error loss landscape of shallow linear neural networks. We show---with significantly milder assumptions than previous works---that the corresponding optimization problems have benign geometric properties: there are no spurious local minima and the Hessian at every saddle point has at least one ne…

In the study of the curve shortening flow on general closed curves, Abresch and Langer posed a conjecture that the homothetic curves can be regarded as saddle points between multi-folded circles and some singular curves. In other words, these homothetic curves are the watershed between curves with a nonsingular future …

A distributed optimization method solves saddle point problems with strong concavity and convexity.

FeDualEx tackles saddle point optimization in federated learning with composite objectives.

Under appropriate cooperation protocols and parameter choices, fully decentralized solutions for stochastic optimization have been shown to match the performance of centralized solutions and result in linear speedup (in the number of agents) relative to non-cooperative approaches in the strongly-convex setting. More re…

In this paper we propose a primal-dual proximal extragradient algorithm to solve the generalized Dantzig selector (GDS) estimation problem, based on a new convex-concave saddle-point (SP) reformulation. Our new formulation makes it possible to adopt recent developments in saddle-point optimization, to achieve the optim…

Saddle-point optimization problems are an important class of optimization problems with applications to game theory, multi-agent reinforcement learning and machine learning. A majority of the rich literature available for saddle-point optimization has focused on the offline setting. In this paper, we study nonstationar…

Gradient descent can take exponentially long to escape saddle points in 2D.

New methods help escape strict saddle points in nonsmooth optimization.

SGD converges almost surely in non-convex problems, avoiding saddle points and accelerating convergence.

A new algorithm trains deep neural networks by adding neurons greedily.

New method solves saddle-point problems faster than existing methods.

A new method helps escape saddle points in non-convex optimization.

Momentum Stochastic Gradient Descent (MSGD) algorithm has been widely applied to many nonconvex optimization problems in machine learning, e.g., training deep neural networks, variational Bayesian inference, and etc. Despite its empirical success, there is still a lack of theoretical understanding of convergence proper…

This paper extends Newton's method to distributed learning, avoiding saddle points and handling Byzantine workers.

Nonconvex optimization algorithms with random initialization have attracted increasing attention recently. It has been showed that many first-order methods always avoid saddle points with random starting points. In this paper, we answer a question: can the nonconvex heavy-ball algorithms with random initialization avoi…

Extends saddle-point method for large-time volatility smiles.

Study geometric properties of loss functions to understand neural network performance.

PWGF escapes saddle points in nonconvex optimization.

Paper defines saddle points in asymmetric Dynkin games using martingale theory.

Gradient-based optimization methods are the most popular choice for finding local optima for classical minimization and saddle point problems. Here, we highlight a systemic issue of gradient dynamics that arise for saddle point problems, namely the presence of undesired stable stationary points that are no local optima…

The deformation problem for pseudoholomorphic curves and related geometrical properties of the total moduli space of pseudoholomorphic curves are studied. A sufficient condition for the saddle point property of the total moduli space is established. The local symplectic isotopy problem is formulated and solved for the …

New algorithm solves saddle point problems in Banach spaces.

We study robust distributed learning that involves minimizing a non-convex loss function with saddle points. We consider the Byzantine setting where some worker machines have abnormal or even arbitrary and adversarial behavior. In this setting, the Byzantine machines may create fake local minima near a saddle point tha…

Although gradient descent (GD) almost always escapes saddle points asymptotically [Lee et al., 2016], this paper shows that even with fairly natural random initialization schemes and non-pathological functions, GD can be significantly slowed down by saddle points, taking exponential time to escape. On the other hand, g…

Local search heuristics for non-convex optimizations are popular in applied machine learning. However, in general it is hard to guarantee that such algorithms even converge to a local minimum, due to the existence of complicated saddle point structures in high dimensions. Many functions have degenerate saddle points su…

DEO uses gradient information to escape saddle points in neural networks.

Loss functions with a large number of saddle points are one of the major obstacles for training modern machine learning models efficiently. First-order methods such as gradient descent are usually the methods of choice for training machine learning models. However, these methods converge to saddle points for certain ch…

Houdini finds high-dimensional saddle points under few constraints.

Gradient-based methods find saddle points, not critical points, in neural networks.

We establish that first-order methods avoid saddle points for almost all initializations. Our results apply to a wide variety of first-order methods, including gradient descent, block coordinate descent, mirror descent and variants thereof. The connecting thread is that such algorithms can be studied from a dynamical s…

New method simplifies optimization landscapes by transforming saddle points.

Paper proves Sion's theorem in geodesic spaces and develops a Riemannian extragradient method.

New method stabilizes saddle-point optimization with unbounded gradients.

We consider variational inequalities coming from monotone operators, a setting that includes convex minimization and convex-concave saddle-point problems. We assume an access to potentially noisy unbiased values of the monotone operators and assess convergence through a compatible gap function which corresponds to the …

Simple gradient descent algorithm escapes saddle points efficiently.

This paper shows that a perturbed form of gradient descent converges to a second-order stationary point in a number iterations which depends only poly-logarithmically on dimension (i.e., it is almost "dimension-free"). The convergence rate of this procedure matches the well-known convergence rate of gradient descent to…

The main result of this paper is: {\bf Theorem.} Let $f:\mathbb{R}^k\rightarrow \mathbb{R}$ be a $C^{1}$ function, so that $\nabla f$ is locally Lipschitz continuous. Assume moreover that $f$ is $C^2$ near its generalised saddle points. Fix real numbers $δ_0>0$ and $0<α<1$. Then there is a smooth function $h:\mathbb{R}…

We extend the Frank-Wolfe (FW) optimization algorithm to solve constrained smooth convex-concave saddle point (SP) problems. Remarkably, the method only requires access to linear minimization oracles. Leveraging recent advances in FW optimization, we provide the first proof of convergence of a FW-type saddle point solv…

In this paper we consider solving saddle point problems using two variants of Gradient Descent-Ascent algorithms, Extra-gradient (EG) and Optimistic Gradient Descent Ascent (OGDA) methods. We show that both of these algorithms admit a unified analysis as approximations of the classical proximal point method for solving…

In this article, we study the convergence of Mirror Descent (MD) and Optimistic Mirror Descent (OMD) for saddle point problems satisfying the notion of coherence as proposed in Mertikopoulos et al. We prove convergence of OMD with exact gradients for coherent saddle point problems, and show that monotone convergence on…

Optimal privacy-preserving algorithm for solving saddle point problems.