arXiv research

While Standard gradient descent is one very popular optimisation method, its convergence cannot be proven beyond the class of functions whose gradient is globally Lipschitz continuous. As such, it is not actually applicable to realistic applications such as Deep Neural Networks. In this paper, we prove that its backtra…

This paper allows unbounded learning rates in gradient descent for better convergence.

Backtracking Gradient Descent method converges to critical points in Banach spaces.

The paper develops methods for unconstrained optimization on Riemannian manifolds.

New method uses robust estimators for Newton's method in empirical risk minimization.

New Q-Newton's method avoids saddle points and converges quadratically.

VAV method optimizes learning rate for faster, stable SGD convergence.

A major drawback of backpropagation through time (BPTT) is the difficulty of learning long-term dependencies, coming from having to propagate credit information backwards through every single step of the forward computation. This makes BPTT both computationally impractical and biologically implausible. For this reason,…

Variance-reduced algorithms, although achieve great theoretical performance, can run slowly in practice due to the periodic gradient estimation with a large batch of data. Batch-size adaptation thus arises as a promising approach to accelerate such algorithms. However, existing schemes either apply prescribed batch-siz…

New matrix reveals cluster info in sparse directed graphs.

NSGD-M optimizes machine learning models without hyperparameter tuning, even under relaxed smoothness.

New approach to counterfactual reasoning in AI and psychology.

This paper presents VEC-NBT, a variation on the unsupervised graph clustering technique VEC, which improves upon the performance of the original algorithm significantly for sparse graphs. VEC employs a novel application of the state-of-the-art word2vec model to embed a graph in Euclidean space via random walks on the n…

DeepBC method computes backtracking counterfactuals in deep causal models.

Paper introduces a privacy-preserving line search method for optimization.

A new graph neural network (NBA-GNN) avoids revisiting nodes to improve accuracy.

A new algorithm improves GAN training stability and performance.

Optimization algorithms with momentum, e.g., (ADAM), have been widely used for building deep learning models due to the faster convergence rates compared with stochastic gradient descent (SGD). Momentum helps accelerate SGD in the relevant directions in parameter updating, which can minify the oscillations of parameter…

Spectral method detects communities in sparse hypergraphs, achieving detection threshold.

Stochastic approximation is one of the effective approach to deal with the large-scale machine learning problems and the recent research has focused on reduction of variance, caused by the noisy approximations of the gradients. In this paper, we have proposed novel variants of SAAG-I and II (Stochastic Average Adjusted…

Blind Descent avoids gradient issues, using a different learning approach.

In many environments only a tiny subset of all states yield high reward. In these cases, few of the interactions with the environment provide a relevant learning signal. Hence, we may want to preferentially train on those high-reward states and the probable trajectories leading to them. To this end, we advocate for the…

Efficient algorithms solve joint graphical lasso problems.

Transformer learns to search through reinforcement learning, mimicking DFS.

Let $z=(x,y)$ be coordinates for the product space $\mathbb{R}^{m_1}\times \mathbb{R}^{m_2}$. Let $f:\mathbb{R}^{m_1}\times \mathbb{R}^{m_2}\rightarrow \mathbb{R}$ be a $C^1$ function, and $\nabla f=(\partial _xf,\partial _yf)$ its gradient. Fix $0<α<1$. For a point $(x,y) \in \mathbb{R}^{m_1}\times \mathbb{R}^{m_2}$, …

In this note, we observe the behavior of gradient flow and discrete and noisy gradient descent in some simple settings. It is commonly noted that addition of noise to gradient descent can affect the trajectory of gradient descent. Here, we run some computer experiments for gradient descent on some simple functions, and…

Stochastic gradient descent on manifolds improves low-rank approximation.

A new method improves stochastic gradient descent for faster and more efficient estimation.

Gradient descent implicitly regularizes neural networks by penalizing large loss gradients.

Derives Mirror Descent from gradient flow on a Riemannian manifold.

We consider the behavior of gradient flow and of discrete and noisy gradient descent. It is commonly noted that the addition of noise to the process of discrete gradient descent can affect the trajectory of gradient descent. In previous work, we observed such effects. There, we considered the case where the minima had …

Gradient descent variants improve phase retrieval accuracy.

Reparameterizes mirror descent as gradient descent for efficient sparse learning.

Develops active intervals for geodesics in Teichmüller space.

Gradient descent dynamics in nonconvex models explained with universality.

Gravilon improves gradient descent for neural networks.

SGD reduces test error by decorrelating updates.

New adaptive step-size method for convex optimization without tuning.

Natural gradient descent avoids the magic of model parametrization, leading to different optimization outcomes.

Any gradient descent optimization requires to choose a learning rate. With deeper and deeper models, tuning that learning rate can easily become tedious and does not necessarily lead to an ideal convergence. We propose a variation of the gradient descent algorithm in the which the learning rate is not fixed. Instead, w…

Gradient descent biases towards stable rank networks for nearly-orthogonal data.

Gradient descent can take exponentially long to escape saddle points in 2D.

SGD and stochastic gradient descent converge at optimal rates for certain non-convex functions.

Gradient descent at edge of stability stabilizes implicitly, following projected gradient descent.

This work proposes a new method for variational inference using Wasserstein gradient descent.

Gradient descent achieves exact linear convergence rate for symmetric matrix completion.

New bounds for model generalization under deterministic gradient descent.

We study the problem of training deep neural networks with Rectified Linear Unit (ReLU) activation function using gradient descent and stochastic gradient descent. In particular, we study the binary classification problem and show that for a broad family of loss functions, with proper random weight initialization, both…