arXiv research

Blind Descent avoids gradient issues, using a different learning approach.

A new method improves stochastic gradient descent for faster and more efficient estimation.

Recent years have seen increased interest in performance guarantees of gradient descent algorithms for non-convex optimization. A number of works have uncovered that gradient noise plays a critical role in the ability of gradient descent recursions to efficiently escape saddle-points and reach second-order stationary p…

AGD converges in polynomial iterations to optimal matrix factorization.

ScaledGD improves gradient descent for ill-conditioned low-rank matrix estimation.

Alt-GDA outperforms Sim-GDA in minimax games with near-optimal local convergence.

This paper proposes an alternating back-propagation algorithm for learning the generator network model. The model is a non-linear generalization of factor analysis. In this model, the mapping from the continuous latent factors to the observed signal is parametrized by a convolutional neural network. The alternating bac…

Paper proposes algorithms for solving nonconvex-nonconcave problems with complexity guarantees.

A new method for SVGD reduces variance in high dimensions.

New method improves MMD estimation without convexity assumptions.

SoftAD improves classification accuracy with less fine-tuning and fewer computational costs.

New method learns good initialization for gradient descent from past solutions.

APGD algorithm efficiently recovers over-parameterized matrices from noisy measurements.

Analytical method finds deeper optima in two-layer ReLU networks.

We introduce a novel and efficient algorithm called the stochastic approximate gradient descent (SAGD), as an alternative to the stochastic gradient descent for cases where unbiased stochastic gradients cannot be trivially obtained. Traditional methods for such problems rely on general-purpose sampling techniques such …

New method improves online covariance estimation for SGD.

The alternating gradient descent (AGD) is a simple but popular algorithm which has been applied to problems in optimization, machine learning, data ming, and signal processing, etc. The algorithm updates two blocks of variables in an alternating manner, in which a gradient step is taken on one block, while keeping the …

This work improves dictionary learning speed without sacrificing accuracy.

Non-negative matrix factorization is a basic tool for decomposing data into the feature and weight matrices under non-negativity constraints, and in practice is often solved in the alternating minimization framework. However, it is unclear whether such algorithms can recover the ground-truth feature matrix when the wei…

This is a report for reproducibility challenge of NeurlIPS 2019 on the paper Competitive Gradient Descent (Schafer et al., 2019). The paper introduces a novel algorithm for the numerical computation of Nash equilibria of competitive two-player games. It avoids oscillatory and divergent behaviours seen in alternating gr…

TSSM splits neural networks for parallel training with minimal accuracy loss.

The paper analyzes the variance of different shuffling methods in stochastic gradient descent.

New algorithm solves federated minimax optimization problems.

Develops DP-SCD for stochastic coordinate descent, making it differentially private.

A new ODE model explains gradient descent dynamics near edge of stability.

The paper explores how symmetries and noise in SGD influence parameter dynamics.

In several experimental reports on nonconvex optimization problems in machine learning, stochastic gradient descent (SGD) was observed to prefer minimizers with flat basins in comparison to more deterministic methods, yet there is very little rigorous understanding of this phenomenon. In fact, the lack of such work has…

Python package for ordinal regression using gradient boosting.

Optimizes neural networks' last layer with closed-form solutions.

Max-affine regression method converges linearly using GD and SGD.

A new method for robust training under label noise using weighted gradient descent.

Adapting functional gradients improves FGD's practicality and theoretical guarantees.

Stein variational gradient descent improves inference in Gaussian process models.

New Muon and Momo variants improve neural network optimization robustness.

A new biased gradient descent method for conditional stochastic optimization.

Momentum based stochastic gradient methods such as heavy ball (HB) and Nesterov's accelerated gradient descent (NAG) method are widely used in practice for training deep networks and other supervised learning models, as they often provide significant improvements over stochastic gradient descent (SGD). Rigorously speak…

We show that accelerated gradient descent, averaged gradient descent and the heavy-ball method for non-strongly-convex problems may be reformulated as constant parameter second-order difference equation algorithms, where stability of the system is equivalent to convergence at rate O(1/n 2), where n is the number of ite…

Ad-SVGD optimizes kernel parameters for SVGD, improving inference performance.

A new catnat function improves gradient descent for categorical variables.

Hybridizes CEM and gradient descent for efficient model-predictive control.

A scalable method for heavy-tailed gradients using cheap stochastic sub-processes.

Paper proposes a new method to separate low rank and sparse matrices without bias.

In this paper, we briefly review the basic scheme of the pseudoinverse learning (PIL) algorithm and present some discussions on the PIL, as well as its variants. The PIL algorithm, first presented in 1995, is a non-gradient descent and non-iterative learning algorithm for multi-layer neural networks and has several adv…

Constructs retractions of CAT(1) spaces to convex subsets.

SimpleMKKM improves multi-kernel clustering efficiency.

Natural gradient descent is an optimization method traditionally motivated from the perspective of information geometry, and works well for many applications as an alternative to stochastic gradient descent. In this paper we critically analyze this method and its properties, and show how it can be viewed as a type of 2…

LES optimizes designs by sampling descent sequences, achieving strong sample efficiency.

A standard introduction to online learning might place Online Gradient Descent at its center and then proceed to develop generalizations and extensions like Online Mirror Descent and second-order methods. Here we explore the alternative approach of putting Exponential Weights (EW) first. We show that many standard meth…