Applying standard Markov chain Monte Carlo (MCMC) algorithms to large data sets is computationally infeasible. The recently proposed stochastic gradient Langevin dynamics (SGLD) method circumvents this problem in three ways: it generates proposed moves using only a subset of the data, it skips the Metropolis-Hastings a…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,738 papers · 148 categories
Trend · papers per month
48 results for “decreasing step-sizes”
Improved Central Limit Theorem and Bootstrap Approximations for Linear Stochastic Approximationstat.ML
Paper improves CLT and bootstrap approximations for LSA with decreasing step size.
problem Improving normal approximation and bootstrap methods for LSA with decreasing step sizes.
method Refined Berry-Esseen bounds and multiplier bootstrap procedure for LSA.
result Approximation rates up to for LSA rescaled error distribution.
Adaptive Importance Sampling for Finite-Sum Optimization and Sampling with Decreasing Step-Sizescs.LG
Avare improves optimization and sampling with adaptive importance sampling.
problem Improving convergence rate of stochastic gradient-based algorithms.
method Adaptive importance sampling with decreasing step-sizes.
result Achieves dynamic regret bounds of and .
The CSA-ES is an Evolution Strategy with Cumulative Step size Adaptation, where the step size is adapted measuring the length of a so-called cumulative path. The cumulative path is a combination of the previous steps realized by the algorithm, where the importance of each step decreases with time. This article studies …
GradaGrad adapts learning rate non-monotonically, overcoming AdaGrad's step size decrease.
problem Fixed learning rate in AdaGrad leads to step size decrease over time.
method Introduces GradaGrad, which grows or shrinks the learning rate based on a different accumulation in the denominator.
result GradaGrad achieves similar convergence rates as AdaGrad and demonstrates non-monotone adaptation.
In an adaptive population which models financial markets and distributed control, we consider how the dynamics depends on the diversity of the agents' initial preferences of strategies. When the diversity decreases, more agents tend to adapt their strategies together. This change in the environment results in dynamical…
Adaptive step-size improves optimization in complex geometries.
problem Optimizing functions with non-Euclidean geometries.
method Adaptive step-size strategy for optimization algorithms.
result Guaranteed convergence for Adaptive Conditional Gradient Descent.
Applying standard Markov chain Monte Carlo (MCMC) algorithms to large data sets is computationally expensive. Both the calculation of the acceptance probability and the creation of informed proposals usually require an iteration through the whole data set. The recently proposed stochastic gradient Langevin dynamics (SG…
SALSA automatically adjusts learning rates in stochastic gradient methods.
problem Automatic adjustment of learning rates in stochastic gradient methods.
method SALSA uses a line-search procedure to gradually increase the learning rate, then a statistical test to decrease it.
result SALSA matches the performance of best hand-tuned learning rate schedules in deep learning tasks.
New convergence results for NGVI with various step sizes and sample sizes.
problem Understanding convergence of stochastic NGVI for various schedules.
method Projected stochastic NGVI for exponential family variational distributions.
result Geometric convergence and $\mathcal{O}\left(\frac{1}{T^ρ}
ight)$ rates for different schedules.
GD monotonically decreases GFS sharpness in neural networks and scalar models.
problem Oscillatory behavior of loss in GD training.
method Analysis of GFS sharpness and empirical validation.
result GFS sharpness decreases monotonically during GD training.
Gaussian Approximation and Multiplier Bootstrap for Federated Linear Stochastic Approximationstat.ML
Paper develops Gaussian approximations and bootstrap for federated LSA with trade-off bounds.
problem Analyzing convergence rates and trade-offs in federated linear stochastic approximation.
method Established Berry-Esseen-type bounds for federated LSA, developed multiplier bootstrap for inference.
result First federated Gaussian approximations with explicit trade-off terms and non-asymptotic validity guarantees.
Characterizes corridors in loss surfaces for gradient-based optimization.
problem Understanding and mitigating training instabilities in gradient-based optimization.
method Characterizes corridors as regions where gradient descent and gradient flow trajectories are linearly related.
result Corridors indicate regions without implicit regularization effects, leading to better learning rate adaptation schemes.
Paper proposes a new method to detect convergence in SGD.
problem Detecting the transition from fast progress to oscillation in SGD.
method Analyzes Pflug's test and proposes a novel statistical procedure.
result The novel procedure accurately detects stationarity in SGD.
Recently it has been shown that the step sizes of a family of variance reduced gradient methods called the JacSketch methods depend on the expected smoothness constant. In particular, if this expected smoothness constant could be calculated a priori, then one could safely set much larger step sizes which would result i…
Deep networks run with low precision operations at inference time offer power and space advantages over high precision alternatives, but need to overcome the challenge of maintaining high accuracy as precision decreases. Here, we present a method for training such networks, Learned Step Size Quantization, that achieves…
Gradually Truncated Log-normal distribution - Size distribution of firmscond-mat.stat-mech
Gradually Truncated Log-normal distribution - Size distribution of firms Abstract Many natural and economical phenomena are described through power law or log- normal distributions. In these cases, probability decreases very slowly with step size compared to normal distribution. Thus it is essential to cut-off these di…
Paper analyzes convergence rates for multi-agent learning in games.
problem Convergence rates for multi-agent learning in games.
method Characterizes finite-time convergence rates for joint OGD learning on -cocoercive games and develops adaptive algorithms.
result Adaptive algorithms achieve same convergence rates as non-adaptive counterparts.
Gradient descent on neural nets often operates at the Edge of Stability, where loss behavior is complex but loss decreases over time.
problem Understanding the optimization dynamics of neural networks at the Edge of Stability.
method Empirical demonstration of gradient descent behavior in neural network training.
result Gradient descent on neural networks typically occurs at the Edge of Stability, where loss behavior is non-monotonic but loss decreases over time.
Recent advances in Bayesian learning with large-scale data have witnessed emergence of stochastic gradient MCMC algorithms (SG-MCMC), such as stochastic gradient Langevin dynamics (SGLD), stochastic gradient Hamiltonian MCMC (SGHMC), and the stochastic gradient thermostat. While finite-time convergence properties of th…
Proposes a semi-implicit back propagation method for neural networks.
problem Challenges in training neural networks, especially gradient vanishing and small step sizes.
method Proposes a semi-implicit back propagation method using error back propagation and proximal methods.
result The proposed method leads to better performance in terms of loss decreasing and training/validation accuracy compared to SGD and ProxBP.
Stochastic Gradient Langevin Dynamics (SGLD) has emerged as a key MCMC algorithm for Bayesian learning from large scale datasets. While SGLD with decreasing step sizes converges weakly to the posterior distribution, the algorithm is often used with a constant step size in practice and has demonstrated successes in mach…
Stagewise training strategy is widely used for learning neural networks, which runs a stochastic algorithm (e.g., SGD) starting with a relatively large step size (aka learning rate) and geometrically decreasing the step size after a number of iterations. It has been observed that the stagewise SGD has much faster conve…
This paper establishes risk convergence and asymptotic weight matrix alignment --- a form of implicit regularization --- of gradient flow and gradient descent when applied to deep linear networks on linearly separable data. In more detail, for gradient flow applied to strictly decreasing loss functions (with similar re…
Although stochastic gradient descent (SGD) method and its variants (e.g., stochastic momentum methods, AdaGrad) are the choice of algorithms for solving non-convex problems (especially deep learning), there still remain big gaps between the theory and the practice with many questions unresolved. For example, there is s…
Decentralized Bayesian learning reduces KL-divergence exponentially.
problem Efficiently learning posterior distributions in a decentralized setting.
method Decentralized Langevin dynamics in a non-convex setting.
result The algorithm converges to the target posterior distribution with exponential decrease in KL-divergence and polynomial decrease in error contributions.
Improved averaging method for noisy observations converges strongly.
problem Noisy observations from random dynamical systems require stable estimates.
method Introduced -EMA, a modified exponential moving average with subharmonic weight decay.
result Stochastic convergence guarantees for -EMA under mild assumptions.
New method improves convergence and reduces variance in noisy optimization problems.
problem Computing exact minimizers with noisy gradient information.
method Stochastic mirror descent with interacting particles.
result Interaction helps improve convergence and reduce variance.
Adapts SGD to noise and problem specifics for faster convergence.
problem Minimizing smooth, strongly-convex functions with varying noise and problem constants.
method Adaptive SGD with exponentially decreasing step-sizes, Nesterov acceleration, and stochastic line-search.
result Achieves near-optimal convergence rates without knowing noise or problem specifics.
Study on RF regression with SGD shows double descent phenomenon.
problem Understanding generalization in RF models trained with SGD.
method Precise non-asymptotic error bounds derived for RF regression under constant and polynomial-decay step-size SGD.
result RF regression generalizes well for interpolation learning and exhibits double descent behavior.
Novel bounds for SGLD show generalization error decreases with more samples.
problem Understanding the generalization error of SGLD in non-convex optimization.
method Information-theoretic approach focusing on Kullback-Leibler divergence and sub-exponential loss function.
result Time-independent generalization bounds for SGLD, independent of step size and number of iterations.
Improved analysis of extragradient methods for structured VIPs.
problem Efficiently solving large-scale VIPs with weaker conditions.
method Single-call stochastic extragradient methods with expected residual condition.
result Convergence guarantees for quasi-strongly monotone and weak Minty VIPs.
Paper improves confidence intervals for LSA with multiplier bootstrap.
problem Improving confidence intervals for parameter estimation in LSA.
method Berry-Esseen bound for multivariate normal approximation and multiplier bootstrap.
result Valid confidence intervals for parameter estimation in LSA.
Implicit Q-learning and SARSA adjust step-sizes automatically, improving stability and performance.
problem Numerical instability and slow progress in Q-learning and SARSA due to step-size calibration.
method Reformulate iterative updates as fixed-point equations, scaling step-sizes inversely with feature norms.
result Implicit methods maintain stability over broader step-size ranges and achieve comparable convergence rates.
The paper analyzes fixed step-size SA schemes on Riemannian manifolds.
problem Developing efficient algorithms for optimization on curved spaces.
method Fixed step-size stochastic approximation schemes in a Riemannian framework.
result The schemes converge to the solution as the step-size approaches zero.
The paper analyzes and validates two step size schedules for SGD: exponential and cosine, proving their adaptivity and performance.
problem The variability of SGD performance due to step size choice.
method Analysis and empirical evaluation of exponential and cosine step sizes.
result Exponential and cosine step sizes are adaptive to noise and achieve optimal performance without tuning hyperparameters.
In this paper, we introduce a method for adapting the step-sizes of temporal difference (TD) learning. The performance of TD methods often depends on well chosen step-sizes, yet few algorithms have been developed for setting the step-size automatically for TD learning. An important limitation of current methods is that…
Improved algorithm reduces regret in corrupted expert advice setting.
problem Prediction with expert advice in the presence of adversarial corruption.
method Multiplicative Weights algorithm with decreasing step sizes.
result Achieves constant regret and optimal performance in various environments.
New algorithm improves stability of optimization algorithms by adapting step-size.
problem Optimization algorithms' effectiveness is sensitive to step-size hyperparameters.
method Adapts NGN step-size method with momentum to enhance stability.
result Achieves convergence rate of O(1/√K) without restrictive assumptions.
SGD converges to global minimum for structured non-convex functions.
problem Optimizing non-convex functions using SGD with slow convergence rates.
method Convergence theorems for SGD on structured non-convex functions, including Quasar and PL conditions.
result SGD converges to global minimum for specific non-convex functions under certain conditions.
AutoStep MCMC adapts step size locally for better sampling efficiency.
problem Challenging step size selection for complex, multiscale targets.
method AutoStep MCMC uses a locally adaptive step size for involutive proposals.
result AutoStep MCMC is π-invariant, irreducible, and aperiodic.
Gradient descent with chaotic perturbations improves generalization.
problem Improving generalization of gradient descent.
method Introducing chaotic perturbations to gradient descent to achieve improved generalization.
result Gradient descent with chaotic perturbations converges to a heavy-tailed SDE, leading to improved generalization.
Negative step sizes improve second-order methods for neural networks.
problem Second-order methods discard negative curvature, limiting their effectiveness.
method Introduce negative step sizes in second-order methods combined with Wolfe line search.
result Negative step sizes lead to global convergence and improved performance.
ContinuousNet generalizes ResNets to continuous dynamical systems.
problem ResNets fail to be meaningful dynamical integrators.
method Embedding continuous dynamical systems into higher-order numerical integration schemes (Runge Kutta).
result ContinuousNet exhibits invariance to discrete time step sizes and numerical integration schemes.
Speed learning on the flymath.OC
The practical performance of online stochastic gradient descent algorithms is highly dependent on the chosen step size, which must be tediously hand-tuned in many applications. The same is true for more advanced variants of stochastic gradients, such as SAGA, SVRG, or AdaGrad. Here we propose to adapt the step size by …
The main goal of this work is equipping convex and nonconvex problems with Barzilai-Borwein (BB) step size. With the adaptivity of BB step sizes granted, they can fail when the objective function is not strongly convex. To overcome this challenge, the key idea here is to bridge (non)convex problems and strongly convex …
New insights into SGD and SGD-M in high dimensions.
problem Understanding and comparing SGD and SGD-M in high-dimensional settings.
method Developed high-dimensional scaling limits for SGD-M and online SGD, examining their dynamics and performance.
result SGD-M amplifies high-dimensional effects, potentially degrading performance compared to online SGD.
Improved variational inequality algorithms using adaptive step sizes.
problem Solving monotone variational inequalities and convex-concave min-max problems efficiently.
method Adaptive step sizes that eliminate hyperparameters and global Lipschitz continuity requirements.
result Eliminated the need for the golden ratio in the algorithm and improved complexity bounds.