arXiv research

Vanilla SGD learns SIM from anisotropic data without explicit covariance estimation.

Fractal learning rate schedules accelerate vanilla gradient descent.

In this paper, we aim at providing an introduction to the gradient descent based optimization algorithms for learning deep neural network models. Deep learning models involving multiple nonlinear projection layers are very challenging to train. Nowadays, most of the deep learning model training still relies on the back…

Stein variational gradient descent (SVGD) is a particle-based inference algorithm that leverages gradient information for efficient approximate inference. In this work, we enhance SVGD by leveraging preconditioning matrices, such as the Hessian and Fisher information matrix, to incorporate geometric information into SV…

In this paper, we provide an overview of first-order and second-order variants of the gradient descent method that are commonly used in machine learning. We propose a general framework in which 6 of these variants can be interpreted as different instances of the same approach. They are the vanilla gradient descent, the…

Improved convergence for overparameterized low-rank matrix sensing.

This work proposes a new method for variational inference using Wasserstein gradient descent.

Noise injection before gradient steps helps in regularization for neural networks.

Two-layer ReLU networks outperform kernel methods in teacher-student settings.

Gradient descent recovers principal components of overparametrized asymmetric matrices without explicit regularization.

This paper proves AdaGrad and Adam converge linearly under PL inequality.

Riemannian gradient descent escapes some spurious critical points on low-rank matrix manifold.

New analysis improves sample complexity for vanilla policy gradient methods.

In this paper, we propose a geometric framework to analyze the convergence properties of gradient descent trajectories in the context of linear neural networks. We translate a well-known empirical observation of linear neural nets into a conjecture that we call the \emph{overfitting conjecture} which states that, for a…

Stochastic gradient descent (\textsc{Sgd}) methods are the most powerful optimization tools in training machine learning and deep learning models. Moreover, acceleration (a.k.a. momentum) methods and diagonal scaling (a.k.a. adaptive gradient) methods are the two main techniques to improve the slow convergence of \text…

This paper considers the problem of solving systems of quadratic equations, namely, recovering an object of interest $\mathbf{x}^{\natural}\in\mathbb{R}^{n}$ from $m$ quadratic equations/samples $y_{i}=(\mathbf{a}_{i}^{\top}\mathbf{x}^{\natural})^{2}$, $1\leq i\leq m$. This problem, also dubbed as phase retrieval, span…

Gradient descent near stability threshold shows sharpness oscillations.

Gradient descent near stability threshold exhibits sharpness oscillations.

New algorithm improves convergence of gradient boosting trees.

Recent years have seen a flurry of activities in designing provably efficient nonconvex procedures for solving statistical estimation problems. Due to the highly nonconvex nature of the empirical loss, state-of-the-art procedures often require proper regularization (e.g. trimming, regularized cost, projection) in order…

New bounds for shallow neural networks with deterministic parameters.

MoE outperforms vanilla networks in detecting latent cluster structures in regression tasks.

This paper investigates different vector step-size adaptation approaches for non-stationary online, continual prediction problems. Vanilla stochastic gradient descent can be considerably improved by scaling the update with a vector of appropriately chosen step-sizes. Many methods, including AdaGrad, RMSProp, and AMSGra…

New method improves optimization algorithms without Lipschitz smoothness.

Polyak's momentum accelerates training of neural networks.

Nonconvex minimax problems appear frequently in emerging machine learning applications, such as generative adversarial networks and adversarial learning. Simple algorithms such as the gradient descent ascent (GDA) are the common practice for solving these nonconvex games and receive lots of empirical success. Yet, it i…

New method reduces communication costs in distributed nonconvex optimization.

Paper characterizes gradient descent dynamics for neural networks with finite width.

The analysis of nonconvex matrix completion has recently attracted much attention in the community of machine learning thanks to its computational convenience. Existing analysis on this problem, however, usually relies on $\ell_{2,\infty}$ projection or regularization that involves unknown model parameters, although th…

Momentum SGD fails to track nonstationary optima due to drift amplification.

Fault-tolerant federated learning for non-uniform data.

Owing to their connection with generative adversarial networks (GANs), saddle-point problems have recently attracted considerable interest in machine learning and beyond. By necessity, most theoretical guarantees revolve around convex-concave (or even linear) problems; however, making theoretical inroads towards effici…

Develops first and second-order pseudo-mirror descent methods for nonnegative function estimation.

Gradient Descent Ascent converges to von-Neumann solution in hidden zero-sum games.

New framework analyzes effectiveness of neural network-based combinatorial problem solvers.

This paper proves SGD converges to global minimum for over-parameterized ReLU networks.

Stochastic Gradient Descent has been widely studied with classification accuracy as a performance measure. However, these stochastic algorithms cannot be directly used when non-decomposable pairwise performance measures are used such as Area under the ROC curve (AUC) which is a common performance metric when the classe…

Stochastic dual coordinate ascent (SDCA) is an effective technique for solving regularized loss minimization problems in machine learning. This paper considers an extension of SDCA under the mini-batch setting that is often used in practice. Our main contribution is to introduce an accelerated mini-batch version of SDC…

Paper studies Adam's convergence under relaxed assumptions, proving a rate of O(poly(log T)/sqrt(T)).

A new transformer model accelerates training with optimization techniques.

We consider the problem of recovering low-rank matrices from random rank-one measurements, which spans numerous applications including covariance sketching, phase retrieval, quantum state tomography, and learning shallow polynomial neural networks, among others. Our approach is to directly estimate the low-rank factor …

Study accelerates optimization methods in non-convex problems, but doesn't improve the algorithm's performance.

Momentum affects optimization differently at small vs large batch sizes near instability.

New algorithm TUSLA improves learning of non-convex neural networks.

SNN architecture shows gradient descent converges to regularized solution in matrix sensing problems.

The paper analyzes GNNs with one hidden layer, proving their generalizability and convergence rate.

Paper introduces SGA for barycenter optimization in optimal transport.

Adversarial training was introduced as a way to improve the robustness of deep learning models to adversarial attacks. This training method improves robustness against adversarial attacks, but increases the models vulnerability to privacy attacks. In this work we demonstrate how model inversion attacks, extracting trai…