A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
WITCHcraft improves PGD attacks with random step size, enhancing efficiency.
Random Function Descent improves optimization in high dimensions.
Study shows how mini-batch GD with random reshuffling affects least squares regression dynamics.
In empirical risk optimization, it has been observed that stochastic gradient implementations that rely on random reshuffling of the data achieve better performance than implementations that rely on sampling the data uniformly. Recent works have pursued justifications for this behavior by examining the convergence rate…
RSGDA improves convergence rates for nonconvex-strongly concave optimization.
We highlight a pitfall when applying stochastic variational inference to general Bayesian networks. For global random variables approximated by an exponential family distribution, natural gradient steps, commonly starting from a unit length step size, are averaged to convergence. This useful insight into the scaling of…
The paper analyzes the randomized midpoint method for Langevin diffusions, revealing biases and asymptotic properties.
The Douglas Rachford algorithm is an algorithm that converges to a minimizer of a sum of two convex functions. The algorithm consists in fixed point iterations involving computations of the proximity operators of the two functions separately. The paper investigates a stochastic version of the algorithm where both funct…
Develops a generalized version of Chung's Lemma for stochastic optimization methods.
We provide larger step-size restrictions for which gradient descent based algorithms (almost surely) avoid strict saddle points. In particular, consider a twice differentiable (non-convex) objective function whose gradient has Lipschitz constant L and whose Hessian is well-behaved. We prove that the probability of init…
Poisson Midpoint Method improves Langevin Dynamics for diffusion models.
StochAstic Recursive grAdient algoritHm (SARAH), originally proposed for convex optimization and also proven to be effective for general nonconvex optimization, has received great attention due to its simple recursive framework for updating stochastic gradient estimates. The performance of SARAH significantly depends o…
Optimizes random forest inference by defining step order to maximize accuracy.
New study shows faster convergence of SGD and Kaczmarz methods.
The weights of a neural network are typically initialized at random, and one can think of the functions produced by such a network as having been generated by a prior over some function space. Studying random networks, then, is useful for a Bayesian understanding of the network evolution in early stages of training. In…
Quantized Stochastic Primal-Dual Methods for Distributed Optimization
SGD batch size affects autoencoder global minima sparsity and sharpness.
Online learners track optimal solutions with constant step-size.
Study on RF regression with SGD shows double descent phenomenon.
Maximal concentration bounds for stochastic approximation with heavy-tailed noise.
Implicit Q-learning and SARSA adjust step-sizes automatically, improving stability and performance.
Data-driven discovery of differential equations has been an emerging research topic. We propose a novel algorithm subsampling-based threshold sparse Bayesian regression (SubTSBR) to tackle high noise and outliers. The subsampling technique is used for improving the accuracy of the Bayesian learning algorithm. It has tw…
The paper investigates model collapse in language models from a probabilistic perspective.
Paper uses Random Matrix Theory for optimal training-testing data split.
We consider the dynamics of a linear stochastic approximation algorithm driven by Markovian noise, and derive finite-time bounds on the moments of the error, i.e., deviation of the output of the algorithm from the equilibrium point of an associated ordinary differential equation (ODE). We obtain finite-time bounds on t…
The paper analyzes fixed step-size SA schemes on Riemannian manifolds.
A new method approximates the exact posterior score for diffusion models.
Sketching and stochastic gradient methods are arguably the most common techniques to derive efficient large scale learning algorithms. In this paper, we investigate their application in the context of nonparametric statistical learning. More precisely, we study the estimator defined by stochastic gradient with mini bat…
The paper analyzes and validates two step size schedules for SGD: exponential and cosine, proving their adaptivity and performance.
NMC improves MCMC convergence by analyzing gradients to determine optimal proposal densities.
The CSA-ES is an Evolution Strategy with Cumulative Step size Adaptation, where the step size is adapted measuring the length of a so-called cumulative path. The cumulative path is a combination of the previous steps realized by the algorithm, where the importance of each step decreases with time. This article studies …
In this paper, we introduce a method for adapting the step-sizes of temporal difference (TD) learning. The performance of TD methods often depends on well chosen step-sizes, yet few algorithms have been developed for setting the step-size automatically for TD learning. An important limitation of current methods is that…
Muons and random optimizers perform similarly, challenging geometric optimization theory.
New algorithm improves stability of optimization algorithms by adapting step-size.
Hamiltonian Monte Carlo (HMC) is a state-of-the-art Markov chain Monte Carlo sampling algorithm for drawing samples from smooth probability densities over continuous spaces. We study the variant most widely used in practice, Metropolized HMC with the Störmer-Verlet or leapfrog integrator, and make two primary contribut…
AutoStep MCMC adapts step size locally for better sampling efficiency.
Adaptive step sizes improve optimization for convex and nonconvex problems.
In this paper we study a family of variance reduction methods with randomized batch size---at each step, the algorithm first randomly chooses the batch size and then selects a batch of samples to conduct a variance-reduced stochastic update. We give the linear convergence rate for this framework for composite functions…
New algorithm approximates maximum of certain distributions on subsets.
This article explores and analyzes the unsupervised clustering of large partially observed graphs. We propose a scalable and provable randomized framework for clustering graphs generated from the stochastic block model. The clustering is first applied to a sub-matrix of the graph's adjacency matrix associated with a re…
Distributed learning with random features and gradient descent improves performance and reduces memory usage.
Polyak step size GD reaches final radius of convergence after log iterations.
Stochastic gradient descent (SGD) on a low-rank factorization is commonly employed to speed up matrix problems including matrix completion, subspace tracking, and SDP relaxation. In this paper, we exhibit a step size scheme for SGD on a low-rank least-squares problem, and we prove that, under broad sampling conditions,…
Random forests have proven to be reliable predictive algorithms in many application areas. Not much is known, however, about the statistical properties of random forests. Several authors have established conditions under which their predictions are consistent, but these results do not provide practical estimates of ran…
Adaptive step-size improves optimization in complex geometries.
Negative step sizes improve second-order methods for neural networks.
New convergence results for NGVI with various step sizes and sample sizes.
Over the last decade, both the neural network and kernel adaptive filter have successfully been used for nonlinear signal processing. However, they suffer from high computational cost caused by their complex/growing network structures. In this paper, we propose two random Euler filters for complex-valued nonlinear filt…