A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Stochastic (sub)gradient methods require step size schedule tuning to perform well in practice. Classical tuning strategies decay the step size polynomially and lead to optimal sublinear rates on (strongly) convex problems. An alternative schedule, popular in nonconvex optimization, is called \emph{geometric step decay…
We consider the least-squares regression problem and provide a detailed asymptotic analysis of the performance of averaged constant-step-size stochastic gradient descent (a.k.a. least-mean-squares). In the strongly-convex case, we provide an asymptotic expansion up to explicit exponentially decaying terms. Our analysis…
Minimax optimal convergence rates for classes of stochastic convex optimization problems are well characterized, where the majority of results utilize iterate averaged stochastic gradient descent (SGD) with polynomially decaying step sizes. In contrast, SGD's final iterate behavior has received much less attention desp…
We provide larger step-size restrictions for which gradient descent based algorithms (almost surely) avoid strict saddle points. In particular, consider a twice differentiable (non-convex) objective function whose gradient has Lipschitz constant L and whose Hessian is well-behaved. We prove that the probability of init…
In this paper, we study the online learning algorithm without explicit regularization terms. This algorithm is essentially a stochastic gradient descent scheme in a reproducing kernel Hilbert space (RKHS). The polynomially decaying step size in each iteration can play a role of regularization to ensure the generalizati…
We consider d-dimensional linear stochastic approximation algorithms (LSAs) with a constant step-size and the so called Polyak-Ruppert (PR) averaging of iterates. LSAs are widely applied in machine learning and reinforcement learning (RL), where the aim is to compute an appropriate θ∗∈Rd (that is a…
Community detection in hypergraphs is explored. Under a generative hypergraph model called "d-wise hypergraph stochastic block model" (d-hSBM) which naturally extends the Stochastic Block Model from graphs to d-uniform hypergraphs, the asymptotic minimax mismatch ratio is characterized. For proving the achievability, w…
AMAGOLD improves stochastic gradient MCMC by infrequent Metropolis-Hastings corrections.
problem Bias in stochastic gradient Hamiltonian Monte Carlo (SGHMC).
method AMAGOLD infrequently uses Metropolis-Hastings corrections to remove bias, with a fixed step size schedule.
result AMAGOLD converges to the target distribution with a fixed, rather than a diminishing, step size, and at most a constant factor slower convergence rate.
A random walk wn on a separable, geodesic hyperbolic metric space X converges to the boundary ∂X with probability one when the step distribution supports two independent loxodromics. In particular, the random walk makes positive linear progress. Progress is known to be linear with exponential decay when …
Stochastic variance reduction algorithms have recently become popular for minimizing the average of a large, but finite, number of loss functions. In this paper, we propose a novel Riemannian extension of the Euclidean stochastic variance reduced gradient algorithm (R-SVRG) to a compact manifold search space. To this e…
In recent years, stochastic variance reduction algorithms have attracted considerable attention for minimizing the average of a large but finite number of loss functions. This paper proposes a novel Riemannian extension of the Euclidean stochastic variance reduced gradient (R-SVRG) algorithm to a manifold search space.…
In this paper, we consider unregularized online learning algorithms in a Reproducing Kernel Hilbert Spaces (RKHS). Firstly, we derive explicit convergence rates of the unregularized online learning algorithms for classification associated with a general gamma-activating loss (see Definition 1 in the paper). Our results…
Stagewise training strategy is widely used for learning neural networks, which runs a stochastic algorithm (e.g., SGD) starting with a relatively large step size (aka learning rate) and geometrically decreasing the step size after a number of iterations. It has been observed that the stagewise SGD has much faster conve…
Improved convergence for actor-critic algorithms in MDPs.
problem Global convergence analysis for actor-critic algorithms in MDPs.
method Introduced an analytical framework to handle complex recursions, established convergence to ε-close globally optimal policy with improved sample complexity.
result Converges to ε-close globally optimal policy with sample complexity of O(ε^(-3)) compared to O(ε^(-2)) for ε-close stationary policy.
Approximations to utility indifference prices are provided for a contingent claim in the large position size limit. Results are valid for general utility functions on the real line and semi-martingale models. It is shown that as the position size approaches infinity, the utility function's decay rate for large negative…
New findings show convergence in SA without square summable step sizes.
problem Finding optimal step sizes for stochastic approximation algorithms.
method Analyzed step-size sequences of the form αn=α0n−ρ with ρ∈(0,1), and derived convergence and rate results.
result Convergence of parameter estimates with probability one and in Lp for any p≥1; MSE rate of convergence is O(αn), improved to O(max{αn2,1/n}) with averaging.
The study introduces anytime learning schedules for large language models without fixed horizons.
problem Training large language models without knowing the total training horizon.
method Theoretical analysis and weight averaging to create anytime learning schedules.
result Theoretical and empirical evidence shows that weight averaging with simple step sizes can achieve comparable final loss to well-tuned cosine schedules.
Stochastic momentum methods trade compute efficiency for serial runtime.
problem Stochastic momentum methods trade compute efficiency for serial runtime.
method Stochastic HB and ASGD for consistent linear regression with Gaussian covariates.
result HB preserves SGD-level CE over a larger batch-size window, allowing larger batches to reduce serial runtime until HB reaches its deterministic accelerated scale.