arXiv research

Step decay schedules improve convergence in non-convex optimization.

Geometric step decay schedules improve stochastic algorithms' convergence on sharp nonconvex problems.

Improved learning rate schedule for least squares regression.

Improved online prediction with guaranteed coverage.

A random walk $w_n$ on a separable, geodesic hyperbolic metric space $X$ converges to the boundary $\partial X$ with probability one when the step distribution supports two independent loxodromics. In particular, the random walk makes positive linear progress. Progress is known to be linear with exponential decay when …

We consider the least-squares regression problem and provide a detailed asymptotic analysis of the performance of averaged constant-step-size stochastic gradient descent (a.k.a. least-mean-squares). In the strongly-convex case, we provide an asymptotic expansion up to explicit exponentially decaying terms. Our analysis…

Paper develops an online learning algorithm for functional data models.

Demon improves neural network training with a decaying momentum approach.

Adaptive LR improves neural network Lipschitz regularity without slowing convergence.

Community detection in hypergraphs is explored. Under a generative hypergraph model called "d-wise hypergraph stochastic block model" (d-hSBM) which naturally extends the Stochastic Block Model from graphs to d-uniform hypergraphs, the asymptotic minimax mismatch ratio is characterized. For proving the achievability, w…

New insights into how to inspect and learn from multi-stage processes and AI reasoning.

New theory sharpens Q-learning with LDTZ rate, proving it's best of both worlds.

SignSGD outperforms SGD in linear regression with optimal scaling laws under PLRF model.

Model proposes neural network for continuous time dynamics with inductive biases.

Improved analysis for fair federated learning reduces dependence on noise floor.

We introduce a novel algorithm that computes the $k$-sparse principal component of a positive semidefinite matrix $A$. Our algorithm is combinatorial and operates by examining a discrete set of special vectors lying in a low-dimensional eigen-subspace of $A$. We obtain provable approximation guarantees that depend on t…

The paper analyzes Teukolsky equations on Kerr backgrounds, proving boundedness and decay of solutions.

In this paper, we study the online learning algorithm without explicit regularization terms. This algorithm is essentially a stochastic gradient descent scheme in a reproducing kernel Hilbert space (RKHS). The polynomially decaying step size in each iteration can play a role of regularization to ensure the generalizati…

AdamNX improves Adam's stability by adjusting its learning rate.

WSD schedule improves model training efficiency by adapting learning rates dynamically.

Gradient descent outperforms ridge regression under certain covariance matrix decay conditions.

Optimal learning rate schedules derived for various tasks.

Gradient descent can use larger step sizes to avoid strict saddle points.

SAD-DPSGD improves model performance on imbalanced medical datasets like HAM10000.

We prove compactification theorems for some complete Kähler manifolds with nonnegative Ricci curvature. Among other things, we prove that a complete noncompact Kähler Ricci flat manifold with maximal volume growth and quadratic curvature decay is a crepant resolution of a normal affine algebraic variety. Furthermore, s…

We develop and analyze efficient "coordinate-wise" methods for finding the leading eigenvector, where each step involves only a vector-vector product. We establish global convergence with overall runtime guarantees that are at least as good as Lanczos's method and dominate it for slowly decaying spectrum. Our methods a…

We study the geometry of infinitely presented groups satisfying the small cancelation condition C'(1/8), and define a standard decomposition (called the criss-cross decomposition) for the elements of such groups. We use it to prove the Rapid Decay property for groups with the stronger small cancelation property C'(1/10…

A new model corrects SBM's bias for power-law degree networks.

Weibull weight-scale parameter $λ$ evolves during AdamW training, with alignment, injection, and decay forces driving its growth and relaxation.

We discover scaling laws for kernel regression loss under various learning rate schedules.

AdamP optimizes momentum-based optimizers for scale-invariant weights, improving model performance.

SA-PEF improves federated learning efficiency by correcting gradient mismatches.

As a step toward understanding the analytic behavior of Type-III Ricci flow singularities, i.e. immortal solutions that exhibit |Rm|<C/t curvature decay, we examine the linearization of an equivalent flow at fixed points discovered recently by Baird--Danielo and Lott: nongradient homogeneous expanding Ricci solitons on…

We consider the least-square linear regression problem with regularization by the $\ell^1$-norm, a problem usually referred to as the Lasso. In this paper, we first present a detailed asymptotic analysis of model consistency of the Lasso in low-dimensional settings. For various decays of the regularization parameter, w…

Gradient descent converges linearly in finite-width networks with positive NTK and compatible conditions.

The paper improves model-based reinforcement learning by using multi-timestep objectives.

Paper analyzes SGD in kernel regression, showing it outperforms offline methods.

Recently, a new multi-step temporal learning algorithm, called $Q(σ)$, unifies $n$-step Tree-Backup (when $σ=0$) and $n$-step Sarsa (when $σ=1$) by introducing a sampling parameter $σ$. However, similar to other multi-step temporal-difference learning algorithms, $Q(σ)$ needs much memory consumption and computation tim…

Study of 4D Ricci solitons with symmetry, finding precise geometric asymptotics.

The study introduces anytime learning schedules for large language models without fixed horizons.

This is the second in a series of three papers in which we initiate the study of very rough solutions to the initial value problem for the Einstein vacuum equations expressed relative to wave coordinates. By very rough we mean solutions which cannot be constructed by the classical techniques of energy estimates and Sob…

We prove boundedness and polynomial decay statements for solutions to the spin $\pm2$ generalized Teukolsky system on a Reissner-Nordström background with small charge. The first equation of the system is the generalization of the standard Teukolsky equation in Schwarzschild for the extreme component of the curvature $…

The paper analyzes reg-SGD for convex problems, proving convergence and quantifying the rate of convergence.

We consider $d$-dimensional linear stochastic approximation algorithms (LSAs) with a constant step-size and the so called Polyak-Ruppert (PR) averaging of iterates. LSAs are widely applied in machine learning and reinforcement learning (RL), where the aim is to compute an appropriate $θ_{*} \in \mathbb{R}^d$ (that is a…

Extends Minkowski stability proof to minimal decay assumptions.

New method creates vacuum data at minimal and borderline decay thresholds.

The paper presents a multi-power law for predicting loss curves across different learning rate schedules.

Study convergence of simulated annealing in continuous and discrete settings.