arXiv research

Step decay schedules improve convergence in non-convex optimization.

Stochastic (sub)gradient methods require step size schedule tuning to perform well in practice. Classical tuning strategies decay the step size polynomially and lead to optimal sublinear rates on (strongly) convex problems. An alternative schedule, popular in nonconvex optimization, is called \emph{geometric step decay…

Improved online prediction with guaranteed coverage.

A random walk $w_n$ on a separable, geodesic hyperbolic metric space $X$ converges to the boundary $\partial X$ with probability one when the step distribution supports two independent loxodromics. In particular, the random walk makes positive linear progress. Progress is known to be linear with exponential decay when …

Minimax optimal convergence rates for classes of stochastic convex optimization problems are well characterized, where the majority of results utilize iterate averaged stochastic gradient descent (SGD) with polynomially decaying step sizes. In contrast, SGD's final iterate behavior has received much less attention desp…

We consider the least-squares regression problem and provide a detailed asymptotic analysis of the performance of averaged constant-step-size stochastic gradient descent (a.k.a. least-mean-squares). In the strongly-convex case, we provide an asymptotic expansion up to explicit exponentially decaying terms. Our analysis…

Paper develops an online learning algorithm for functional data models.

Adaptive LR improves neural network Lipschitz regularity without slowing convergence.

Community detection in hypergraphs is explored. Under a generative hypergraph model called "d-wise hypergraph stochastic block model" (d-hSBM) which naturally extends the Stochastic Block Model from graphs to d-uniform hypergraphs, the asymptotic minimax mismatch ratio is characterized. For proving the achievability, w…

Momentum is a widely used technique for gradient-based optimizers in deep learning. In this paper, we propose a decaying momentum (\textsc{Demon}) rule. We conduct the first large-scale empirical analysis of momentum decay methods for modern neural network optimization, in addition to the most popular learning rate dec…

New insights into how to inspect and learn from multi-stage processes and AI reasoning.

New theory sharpens Q-learning with LDTZ rate, proving it's best of both worlds.

SignSGD outperforms SGD in linear regression with optimal scaling laws under PLRF model.

Model proposes neural network for continuous time dynamics with inductive biases.

Improved analysis for fair federated learning reduces dependence on noise floor.

We introduce a novel algorithm that computes the $k$-sparse principal component of a positive semidefinite matrix $A$. Our algorithm is combinatorial and operates by examining a discrete set of special vectors lying in a low-dimensional eigen-subspace of $A$. We obtain provable approximation guarantees that depend on t…

The paper analyzes Teukolsky equations on Kerr backgrounds, proving boundedness and decay of solutions.

In this paper, we study the online learning algorithm without explicit regularization terms. This algorithm is essentially a stochastic gradient descent scheme in a reproducing kernel Hilbert space (RKHS). The polynomially decaying step size in each iteration can play a role of regularization to ensure the generalizati…

AdamNX improves Adam's stability by adjusting its learning rate.

WSD schedule improves model training efficiency by adapting learning rates dynamically.

Gradient descent outperforms ridge regression under certain covariance matrix decay conditions.

Optimal learning rate schedules derived for various tasks.

We provide larger step-size restrictions for which gradient descent based algorithms (almost surely) avoid strict saddle points. In particular, consider a twice differentiable (non-convex) objective function whose gradient has Lipschitz constant L and whose Hessian is well-behaved. We prove that the probability of init…

SAD-DPSGD improves model performance on imbalanced medical datasets like HAM10000.

We prove compactification theorems for some complete Kähler manifolds with nonnegative Ricci curvature. Among other things, we prove that a complete noncompact Kähler Ricci flat manifold with maximal volume growth and quadratic curvature decay is a crepant resolution of a normal affine algebraic variety. Furthermore, s…

We develop and analyze efficient "coordinate-wise" methods for finding the leading eigenvector, where each step involves only a vector-vector product. We establish global convergence with overall runtime guarantees that are at least as good as Lanczos's method and dominate it for slowly decaying spectrum. Our methods a…

We study the geometry of infinitely presented groups satisfying the small cancelation condition C'(1/8), and define a standard decomposition (called the criss-cross decomposition) for the elements of such groups. We use it to prove the Rapid Decay property for groups with the stronger small cancelation property C'(1/10…

Weibull weight-scale parameter $λ$ evolves during AdamW training, with alignment, injection, and decay forces driving its growth and relaxation.

We discover scaling laws for kernel regression loss under various learning rate schedules.

AdamP optimizes momentum-based optimizers for scale-invariant weights, improving model performance.

SA-PEF improves federated learning efficiency by correcting gradient mismatches.

As a step toward understanding the analytic behavior of Type-III Ricci flow singularities, i.e. immortal solutions that exhibit |Rm|<C/t curvature decay, we examine the linearization of an equivalent flow at fixed points discovered recently by Baird--Danielo and Lott: nongradient homogeneous expanding Ricci solitons on…

We consider the least-square linear regression problem with regularization by the $\ell^1$-norm, a problem usually referred to as the Lasso. In this paper, we first present a detailed asymptotic analysis of model consistency of the Lasso in low-dimensional settings. For various decays of the regularization parameter, w…

Stochastic block models (SBMs) have been playing an important role in modeling clusters or community structures of network data. But, it is incapable of handling several complex features ubiquitously exhibited in real-world networks, one of which is the power-law degree characteristic. To this end, we propose a new var…

Gradient descent converges linearly in finite-width networks with positive NTK and compatible conditions.

The paper improves model-based reinforcement learning by using multi-timestep objectives.

Recently, a new multi-step temporal learning algorithm, called $Q(σ)$, unifies $n$-step Tree-Backup (when $σ=0$) and $n$-step Sarsa (when $σ=1$) by introducing a sampling parameter $σ$. However, similar to other multi-step temporal-difference learning algorithms, $Q(σ)$ needs much memory consumption and computation tim…

Study of 4D Ricci solitons with symmetry, finding precise geometric asymptotics.

Paper analyzes SGD in kernel regression, showing it outperforms offline methods.

The study introduces anytime learning schedules for large language models without fixed horizons.

This is the second in a series of three papers in which we initiate the study of very rough solutions to the initial value problem for the Einstein vacuum equations expressed relative to wave coordinates. By very rough we mean solutions which cannot be constructed by the classical techniques of energy estimates and Sob…

We prove boundedness and polynomial decay statements for solutions to the spin $\pm2$ generalized Teukolsky system on a Reissner-Nordström background with small charge. The first equation of the system is the generalization of the standard Teukolsky equation in Schwarzschild for the extreme component of the curvature $…

Extends Minkowski stability proof to minimal decay assumptions.

New method creates vacuum data at minimal and borderline decay thresholds.

The paper analyzes reg-SGD for convex problems, proving convergence and quantifying the rate of convergence.

We consider $d$-dimensional linear stochastic approximation algorithms (LSAs) with a constant step-size and the so called Polyak-Ruppert (PR) averaging of iterates. LSAs are widely applied in machine learning and reinforcement learning (RL), where the aim is to compute an appropriate $θ_{*} \in \mathbb{R}^d$ (that is a…

The paper presents a multi-power law for predicting loss curves across different learning rate schedules.

Unique solutions found for wave-like decaying null infinity equations.