arXiv research

LES optimizes designs by sampling descent sequences, achieving strong sample efficiency.

In this paper we elaborate a general homotopy-theoretic framework in which to study problems of descent and completion and of their duals, codescent and cocompletion. Our approach to homotopic (co)descent and to derived (co)completion can be viewed as $\infty$-category-theoretic, as our framework is constructed in the …

Hybridizes CEM and gradient descent for efficient model-predictive control.

Deep neural networks struggle with numerical instability during training.

Transformers learn causal structure through gradient descent on self-attention mechanisms.

Paper proposes efficient algorithms for designing SLOPE penalty sequences.

When samples have internal structure, we often see a mismatch between the objective optimized during training and the model's goal during inference. For example, in sequence-to-sequence modeling we are interested in high-quality translated sentences, but training typically uses maximum likelihood at the word level. The…

Gradient equilibrium improves online learning performance without requiring sublinear regret.

Transformers can efficiently approximate nonparametric regression with minimal parameters and sequences.

Overparameterization helps prevent forgetting in sequential learning tasks.

The study explores how Transformers predict the next token in a sequence.

We consider the minimization of a function defined on a Riemannian manifold $\mathcal{M}$ accessible only through unbiased estimates of its gradients. We develop a geometric framework to transform a sequence of slowly converging iterates generated from stochastic gradient descent (SGD) on $\mathcal{M}$ to an averaged i…

Study on neural networks forgetting in continual learning.

We study the problem of training deep neural networks with Rectified Linear Unit (ReLU) activation function using gradient descent and stochastic gradient descent. In particular, we study the binary classification problem and show that for a broad family of loss functions, with proper random weight initialization, both…

Develops anytime-valid stopping rules for SGD based on observed trajectory.

In this paper, we study the stochastic gradient descent (SGD) method for the nonconvex nonsmooth optimization, and propose an accelerated SGD method by combining the variance reduction technique with Nesterov's extrapolation technique. Moreover, based on the local error bound condition, we establish the linear converge…

sgdGMF efficiently estimates generalized matrix factorization models for single-cell RNA sequencing data.

Accelerated coordinate descent is widely used in optimization due to its cheap per-iteration cost and scalability to large-scale problems. Up to a primal-dual transformation, it is also the same as accelerated stochastic gradient descent that is one of the central methods used in machine learning. In this paper, we imp…

We prove that stochastic gradient descent efficiently converges to the global optimizer of the maximum likelihood objective of an unknown linear time-invariant dynamical system from a sequence of noisy observations generated by the system. Even though the objective function is non-convex, we provide polynomial running …

Given a sequential learning algorithm and a target model, sequential machine teaching aims to find the shortest training sequence to drive the learning algorithm to the target model. We present the first principled way to find such shortest training sequences. Our key insight is to formulate sequential machine teaching…

Study SGD dynamics in sequence models, revealing training phases and influence of sequence length.

By lifting the ReLU function into a higher dimensional space, we develop a smooth multi-convex formulation for training feed-forward deep neural networks (DNNs). This allows us to develop a block coordinate descent (BCD) training algorithm consisting of a sequence of numerically well-behaved convex optimizations. Using…

New method speeds up change-point detection in data sequences.

The paper analyzes how GANs converge using dual metric flows.

In this paper, we propose a unified view of gradient-based algorithms for stochastic convex composite optimization by extending the concept of estimate sequence introduced by Nesterov. More precisely, we interpret a large class of stochastic optimization methods as procedures that iteratively minimize a surrogate of th…

SVGD algorithm converges at rate 1/sqrt(log log n) for sub-Gaussian distributions.

Improved privacy analysis for stochastic gradient descent.

Transformers learn to recall with non-orthogonal embeddings in realistic settings.

Deep Neural Network (DNN) acoustic models often use discriminative sequence training that optimises an objective function that better approximates the word error rate (WER) than frame-based training. Sequence training is normally implemented using Stochastic Gradient Descent (SGD) or Hessian Free (HF) training. This pa…

We investigate a sequential optimization procedure to minimize the empirical risk functional $f_{\hatθ}(x) = \frac{1}{2}\|G_{\hatθ}(x) - y\|^2$ for certain families of deep networks $G_θ(x)$. The approach is to optimize a sequence of objective functions that use network parameters obtained during different stages of th…

Double descent observed in tree-based models for genomic prediction.

Neural network optimizes learning sequence for reading words.

New method accelerates steepest descent for convex optimization.

In this paper we consider online mirror descent (OMD) algorithms, a class of scalable online learning algorithms exploiting data geometric structures through mirror maps. Necessary and sufficient conditions are presented in terms of the step size sequence $\{η_t\}_{t}$ for the convergence of an OMD algorithm with respe…

This paper describes a new online convex optimization method which incorporates a family of candidate dynamical models and establishes novel tracking regret bounds that scale with the comparator's deviation from the best dynamical model in this family. Previous online optimization methods are designed to have a total a…

New method tracks time-varying parameters in data.

VRSMD improves SMD convergence and has implicit regularization.

We study online convex optimization under stochastic sub-gradient observation faults, where we introduce adaptive algorithms with minimax optimal regret guarantees. We specifically study scenarios where our sub-gradient observations can be noisy or even completely missing in a stochastic manner. To this end, we propose…

Random extrapolation speeds up coordinate descent for sparse and dense data.

A new optimizer, Grad-Avg, converges faster than SGD and improves classification task performance.

New method finds linear relationships across multiple data blocks using proximal gradient descent with $\ell_1$ constraint.

In deterministic optimization, line searches are a standard tool ensuring stability and efficiency. Where only stochastic gradients are available, no direct equivalent has so far been formulated, because uncertain gradients do not allow for a strict sequence of decisions collapsing the search space. We construct a prob…

In deterministic optimization, line searches are a standard tool ensuring stability and efficiency. Where only stochastic gradients are available, no direct equivalent has so far been formulated, because uncertain gradients do not allow for a strict sequence of decisions collapsing the search space. We construct a prob…

Gradient descent optimally trains RNNs without overparameterization.

The paper analyzes the variance of different shuffling methods in stochastic gradient descent.

Recursive least-squares algorithms often use forgetting factors as a heuristic to adapt to non-stationary data streams. The first contribution of this paper rigorously characterizes the effect of forgetting factors for a class of online Newton algorithms. For exp-concave and strongly convex objectives, the algorithms a…

Improves adaptivity in sequence models by over-parameterizing.

Deep networks learn hierarchical functions more efficiently than shallow ones.