arXiv research

Adaptive methods improve gradient descent and proximal gradient for convex optimization.

SGD and stochastic gradient descent converge at optimal rates for certain non-convex functions.

Gradient descent learns useful features even in the NTK regime.

Local Gradient Descent with local steps converges to the centralized model in the interpolation regime.

We provide the first convergence analysis of local gradient descent for minimizing the average of smooth and convex but otherwise arbitrary functions. Problems of this form and local gradient descent as a solution method are of importance in federated learning, where each function is based on private data stored by a u…

Gradient descent achieves exact linear convergence rate for symmetric matrix completion.

We show that gradient descent converges to a local minimizer, almost surely with random initialization. This is proved by applying the Stable Manifold Theorem from dynamical systems theory.

New adaptive step-size method for convex optimization without tuning.

Gradient descent fails to train two-layer ReLU networks, leading to poor performance.

MSGD outperforms SGD in overparametrized settings with faster convergence rates.

Gradient descent-ascent converges to strict local minmax equilibria with a finite timescale separation.

Preconditioned non-convex gradient descent improves noisy matrix estimation.

Stochastic gradient descent optimizes Nyström samples for kernel matrix approximation.

Gradient descent converges linearly for overparameterized linear networks.

Analytical method finds deeper optima in two-layer ReLU networks.

In this paper, we propose a simple, fast and easy to implement algorithm LOSSGRAD (locally optimal step-size in gradient descent), which automatically modifies the step-size in gradient descent during neural networks training. Given a function $f$, a point $x$, and the gradient $\nabla_x f$ of $f$, we aim to find the s…

Gradient descent can take exponentially long to escape saddle points in 2D.

Paper proposes a pre-conditioning technique to speed up gradient-descent convergence in distributed linear least-squares problems.

Paper proposes a pre-conditioning method to speed up gradient descent in multi-agent optimization.

We consider the problem of learning a one-hidden-layer neural network with non-overlapping convolutional layer and ReLU activation, i.e., $f(\mathbf{Z}, \mathbf{w}, \mathbf{a}) = \sum_j a_jσ(\mathbf{w}^T\mathbf{Z}_j)$, in which both the convolutional weights $\mathbf{w}$ and the output weights $\mathbf{a}$ are paramete…

Algorithm improves online canonical correlation analysis.

Stochastic gradient descent is a simple approach to find the local minima of a cost function whose evaluations are corrupted by noise. In this paper, we develop a procedure extending stochastic gradient descent algorithms to the case where the function is defined on a Riemannian manifold. We prove that, as in the Eucli…

We study the problem of training deep neural networks with Rectified Linear Unit (ReLU) activation function using gradient descent and stochastic gradient descent. In particular, we study the binary classification problem and show that for a broad family of loss functions, with proper random weight initialization, both…

KSD Descent uses KSD to sample from a target distribution efficiently.

Accelerates ERM problems with LPI-GD and improved oracle complexity.

Recent work has argued that stochastic gradient descent can approximate the Bayesian uncertainty in model parameters near local minima. In this work we develop a similar correspondence for minibatch natural gradient descent (NGD). We prove that for sufficiently small learning rates, if the model predictions on the trai…

Heavy-tailed distributions emerge in SGD's parameter evolution.

We design a non-convex second-order optimization algorithm that is guaranteed to return an approximate local minimum in time which scales linearly in the underlying dimension and the number of training examples. The time complexity of our algorithm to find an approximate local minimum is even faster than that of gradie…

Proposes a continuous flow model to understand and control instability in gradient descent for deep learning.

Despite the growing prominence of generative adversarial networks (GANs), optimization in GANs is still a poorly understood topic. In this paper, we analyze the "gradient descent" form of GAN optimization i.e., the natural setting where we simultaneously take small gradient steps in both generator and discriminator par…

We analyze stochastic gradient descent for optimizing non-convex functions. In many cases for non-convex functions the goal is to find a reasonable local minimum, and the main concern is that gradient updates are trapped in saddle points. In this paper we identify strict saddle property for non-convex problem that allo…

Proposes Coherent Gradients to explain and reduce overfitting in neural networks.

We study the implicit bias of gradient descent methods in solving a binary classification problem over a linearly separable dataset. The classifier is described by a nonlinear ReLU model and the objective function adopts the exponential loss function. We first characterize the landscape of the loss function and show th…

We present a strikingly simple proof that two rules are sufficient to automate gradient descent: 1) don't increase the stepsize too fast and 2) don't overstep the local curvature. No need for functional values, no line search, no information about the function except for the gradients. By following these rules, you get…

A new method compresses large datasets for gradient descent algorithms efficiently.

Gradient descent converges linearly for neural networks with specific conditions.

We develop a more efficient NGD method for structured parameters.

ECD algorithm speeds up non-convex optimization, offering quantum and stochastic enhancements.

SGD converges with positive probability for non-convex deep neural networks under specific conditions.

Study local convergence of GDA for training GANs with kernel-based discriminators.

We prove the local convergence to minima and estimates on the rate of convergence for the stochastic gradient descent method in the case of not necessarily globally convex nor contracting objective functions. In particular, the results are applicable to simple objective functions arising in machine learning.

New algorithm corrects bias in LDP-released data for better analysis.

SGD converges to global minimum for certain non-convex functions.

LES optimizes designs by sampling descent sequences, achieving strong sample efficiency.

NGD models have higher effective dimension than SGD models.

A new algorithm solves sparse optimization problems on measures efficiently.

Paper proposes a federated learning method for quantile inference with local differential privacy.

Gradient descent at edge of stability stabilizes implicitly, following projected gradient descent.