Study on stochastic approximation with Polyak-Ruppert averaging for linear systems.
problem Understanding the asymptotic and non-asymptotic properties of stochastic approximation procedures.
method Detailed analysis of linear stochastic approximation with Polyak-Ruppert averaging, focusing on asymptotic and non-asymptotic properties.
result Proves CLT and non-asymptotic concentration inequality for averaged iterates, providing refined understanding of linear stochastic approximation.
Polyak-Ruppert CLT for SA-Adam with momentum and non-convergent adaptive preconditioning
problem Adaptive optimizers combining momentum and non-convergent preconditioning
method Proving positive drift stability and a non-autonomous Polyak-Ruppert CLT for SA-Adam
result The iterate-marginal covariance is exactly the plain stochastic gradient descent (SGD) sandwich
Study Q-learning with averaging for reinforcement learning, proving efficient inference and error bounds.
problem Efficient inference and error bounds for Q-learning with averaging.
method Functional central limit theorem and asymptotic linear estimator for optimal Q-value function.
result Standardized partial-sum process converges weakly to a rescaled Brownian motion, matching instance-dependent lower bound for error.
Paper proves CLTs for Q-learning with asynchronous updates.
problem Establishing convergence rates for Q-learning algorithms.
method Polyak-Ruppert averaging, non-asymptotic and functional CLTs.
result Convergence rates in Wasserstein distance for Q-learning.
TD(0) with Polyak-Ruppert averaging achieves robust and fast convergence rates
problem TD(0) learning under Markovian sampling
method Polyak-Ruppert averaging with a single stepsize
result Simultaneous high-probability convergence guarantees for TD(0) iterates and PR average
The paper analyzes time-dependent streaming data with biased gradient estimates and proposes improved stochastic optimization methods.
problem Stochastic optimization in a streaming setting with time-dependent and biased gradient estimates.
method Analysis of several first-order methods including SGD, mini-batch SGD, and time-varying mini-batch SGD, along with their Polyak-Ruppert averages.
result Time-varying mini-batch SGD methods can break long- and short-range dependence structures, and biased SGD methods can achieve comparable performance to their unbiased counterparts.
New streaming methods improve convergence rates for optimization problems.
problem Optimizing large-scale, sequential data problems.
method Time-varying mini-batches and Polyak-Ruppert averaging for gradient-based algorithms.
result Time-varying mini-batches and averaging achieve optimal convergence and variance reduction.
Paper improves confidence intervals for LSA with multiplier bootstrap.
problem Improving confidence intervals for parameter estimation in LSA.
method Berry-Esseen bound for multivariate normal approximation and multiplier bootstrap.
result Valid confidence intervals for parameter estimation in LSA.
Paper analyzes normal approximation for two-timescale stochastic algorithms, revealing interaction between fast and slow timescales.
problem Non-asymptotic bounds for accuracy of normal approximation in linear two-timescale stochastic approximation algorithms.
method Established bounds for normal approximation in terms of convex distance, focusing on last iterate and Polyak-Ruppert averaging.
result Normal approximation rate for the last iterate improves with increased timescale separation, while it decreases in the averaged setting.
We propose and analyze a variant of the classic Polyak-Ruppert averaging scheme, broadly used in stochastic gradient methods. Rather than a uniform average of the iterates, we consider a weighted average, with weights decaying in a geometric fashion. In the context of linear least squares regression, we show that this …
The paper explores how dynamic preconditioning affects the CLT in online averaging.
problem When does dynamic preconditioning preserve the Polyak-Ruppert CLT?
method The authors decompose the averaged error and identify a stabilization-rate threshold for the CLT to hold.
result The CLT holds if the dynamic remainder vanishes in L 2 L^2 L 2 and the stabilization rate exceeds a threshold. The paper offers precise bounds for averaged LSA iterates in linear systems.
problem Computing approximate solutions of linear systems with noisy observations.
method Finite-time analysis of LSA algorithms with Polyak-Ruppert averaging.
result Sharp high-probability bounds for averaged LSA iterates.
New method for zeroth-order stochastic gradient algorithms provides confidence intervals.
problem Lack of inferential capabilities for zeroth-order stochastic gradient algorithms.
method Established central limit theorem and provided online estimators for asymptotic covariance matrix.
result Asymptotically valid confidence sets for parameter estimation and prediction.
Paper develops bounds for stochastic approximation with averaging.
problem Establish high-probability bounds for averaged stochastic approximation.
method Develops a general framework for non-asymptotic concentration bounds.
result Derives sharp bounds for averaged iterates and tightens existing results.
Paper develops methods for statistical inference in SGD with infinite variance.
problem Challenges in statistical inference for SGD with infinite variance.
method Model-agnostic methodology based on weak convergence and subsampling calibration.
result Asymptotically valid confidence regions for SGD in both finite and infinite variance regimes.
In this paper, we study a simple algorithm to construct asymptotically valid confidence regions for model parameters using the batch means method. The main idea is to cancel out the covariance matrix which is hard/costly to estimate. In the process of developing the algorithm, we establish process-level functional cent…
Study optimizes decisions in real-time using inexact simulation solutions.
problem Real-time decision-making in simulation optimization with inexact solutions.
method Optimize then predict (OTP) approach, analyzing bias and variance in simulation-optimization algorithms.
result Unified analysis framework for OTP, establishing convergence rates and optimal allocation of computational budget.
Paper analyzes LSA algorithm bias and error bounds with RR extrapolation.
problem Analyzing bias and high-order error bounds of LSA with Markovian noise.
method Polyak-Ruppert averaging, linearization, Richardson-Romberg extrapolation.
result RR extrapolation effectively cancels the leading bias term.
We provide non-asymptotic convergence rates of the Polyak-Ruppert averaged stochastic gradient descent (SGD) to a normal random vector for a class of twice-differentiable test functions. A crucial intermediate step is proving a non-asymptotic martingale central limit theorem (CLT), i.e., establishing the rates of conve…
New convergence rates for SGD under heavy-tailed noise with infinite variance.
problem Convergence analysis of SGD under heavy-tailed noise with infinite variance.
method Identifying a condition on the Hessian and providing a convergence rate for the distance to the global optimum.
result SGD can converge to the global optimum under heavy-tailed noise with infinite variance.
Smoothed SGD improves quantile estimation without crossing curves.
problem Estimating quantiles without crossing estimated curves.
method Smoothed SGD algorithm with Bahadur representation and Gaussian approximation.
result Smoothed SGD provides non-asymptotic tail probability bounds and a Gaussian approximation for quantile estimates.
Paper improves CLT and bootstrap approximations for LSA with decreasing step size.
problem Improving normal approximation and bootstrap methods for LSA with decreasing step sizes.
method Refined Berry-Esseen bounds and multiplier bootstrap procedure for LSA.
result Approximation rates up to 1 / n 1/\sqrt{n} 1/ n for LSA rescaled error distribution. Stochastic gradient descent's long-term fluctuations are described by a diffusion limit.
problem Long-term behavior of stochastic gradient descent in non-smooth settings.
method Functional central limit theorem applied to rescaled trajectory of SGD.
result Characterization of long-term fluctuations around the minimizer.
Paper improves TD learning algorithm bounds with linear approx.
problem Sharp bounds for TD method performance in MDPs.
method Polyak-Ruppert averaging, universal step size, refined error bounds, stability of random matrices.
result Near-optimal variance and bias terms achieved.
Paper develops online statistical inference methods for stochastic optimization using Kiefer-Wolfowitz algorithms.
problem Online statistical inference of model parameters in stochastic optimization problems.
method Kiefer-Wolfowitz algorithm with random search directions, asymptotic distribution analysis.
result Developed valid confidence intervals for online statistical inference.
New methods improve temporal difference learning for policy evaluation in Markov decision processes.
problem Improving temporal difference learning for policy evaluation in Markov decision processes.
method Introduced variance-reduced forms of stochastic approximation to achieve non-asymptotic, instance-dependent optimality.
result Temporal difference learning is strictly suboptimal, but variance-reduced forms achieve optimality up to logarithmic factors.
Study on bias of constant-step stochastic approximation with Markovian noise.
problem Understanding the bias in stochastic approximation algorithms with Markovian noise.
method Infinitesimal generator comparisons to analyze bias, Lyapunov equation for time-averaged bias, Richardson-Romberg extrapolation for bias reduction.
result Bias of the algorithm is of order O ( α ) O(α) O ( α ) and time-averaged bias is α V + O ( α 2 ) αV + O(α^2) α V + O ( α 2 ) , where V V V is a constant. Paper derives convergence rates and confidence intervals for LSA with Markovian noise.
problem Analyzing convergence rates and constructing confidence intervals for LSA with Markovian noise.
method Derives non-asymptotic Berry-Esseen bounds and multiplier block bootstrap procedure.
result Provides O ( n − 1 / 4 ) \mathcal{O}(n^{-1/4}) O ( n − 1/4 ) convergence rates and guarantees consistent inference. We consider d d d -dimensional linear stochastic approximation algorithms (LSAs) with a constant step-size and the so called Polyak-Ruppert (PR) averaging of iterates. LSAs are widely applied in machine learning and reinforcement learning (RL), where the aim is to compute an appropriate θ ∗ ∈ R d θ_{*} \in \mathbb{R}^d θ ∗ ∈ R d (that is a…
New theory sharpens Q-learning with LDTZ rate, proving it's best of both worlds.
problem Improving Q-learning's theoretical and practical performance.
method Developed a sharp non-asymptotic error bound and central limit theory for Q-learning with PD2Z-ν schedule.
result Q-learning with LDTZ schedule achieves rapid decay and asymptotic convergence guarantees.
The paper analyzes Q-learning convergence rates with asynchronous updates.
problem Analyzing convergence rates of asynchronous Q-learning algorithms.
method Derives rates of convergence using high-dimensional central limit theorems.
result Establishes a rate of order up to n − 1 / 6 log 4 ( n S A ) n^{-1/6} \log^{4} (nS A) n − 1/6 log 4 ( n S A ) for hyper-rectangles. New method for online inference using SGD with random scaling.
problem Efficient online inference for SGD parameters.
method Asymptotic pivotal statistics via random scaling.
result Robust and efficient online inference without resampling.
A method for efficient statistical inference from online algorithms.
problem Computational constraints in online algorithms make traditional variance estimation difficult.
method HulC method that wraps around online algorithms to produce valid confidence regions.
result The HulC method produces asymptotically valid confidence regions for online algorithms.
SGD converges to an invariant distribution with sub-Gaussian or sub-exponential properties.
problem Optimizing smooth and strongly convex objectives using SGD.
method Analysis through Markov chains, focusing on convergence and concentration properties.
result SGD iterates and their invariant limit distribution inherit sub-Gaussian or sub-exponential concentration properties.
The paper analyzes two ISGD modes for statistical inference, deriving error bounds and confidence intervals.
problem Statistical inference with implicit SGD for smooth convex functions.
method Proximal Robbins-Monro (proxRM) and proximal Polyak-Ruppert (proxPR) procedures for ISGD.
result Derives non-asymptotic error bounds and confidence interval estimators for model parameters.
The paper analyzes the convergence rates of Q-learning with entropy regularization and linear function approximation.
problem Analyzing the convergence rates of Q-learning with entropy regularization and linear function approximation.
method The paper derives rates of convergence using the high-dimensional central limit theorem, linearization of the soft Bellman recursion, and Gaussian approximation for the leading martingale term.
result The algorithm's last iterate satisfies high-order moment bounds, with a Gaussian approximation bound of order n − 1 / 4 n^{-1/4} n − 1/4 . New statistical methods improve TD learning for policy evaluation.
problem Improving statistical inference for reinforcement learning.
method Polyak-Ruppert averaging, refined high-dimensional Berry-Esseen bounds, online plug-in estimator, asymptotic covariance matrix.
result Guaranteed finite-sample coverage of confidence regions and simultaneous confidence intervals.
Locally private online quantile regression method addresses privacy constraints.
problem Estimating and inferring quantile regression under local differential privacy constraints.
method Developed a finite-alphabet channel where users compute local contributions, apply randomized response, and send reports. A public decoder corrects distortion and reconstructs inputs for averaging.
result Established local privacy, decoder unbiasedness, consistency, asymptotic normality, and inference for scalar contrasts.
The paper analyzes SGD with Richardson-Romberg extrapolation for convex optimization problems.
problem Solving strongly convex and smooth minimization problems efficiently.
method Combining SGD with Polyak-Ruppert averaging and Richardson-Romberg extrapolation.
result An expansion of the mean-squared error of the estimator with respect to the number of iterations.
Study optimizes solving fixed-point equations using subspace search.
problem Solving linear fixed point equations in Hilbert spaces.
method Linear stochastic approximation scheme with Polyak--Ruppert averaging.
result Established optimal approximation factor for temporal difference learning methods.
Artemis framework improves distributed learning with bidirectional compression and partial participation.
problem Learning in distributed or federated settings with communication constraints and device partial participation.
method Artemis framework using bidirectional compression, memory mechanism, and Polyak-Ruppert averaging.
result Fast rates of convergence (linear up to a threshold) under weak assumptions on stochastic gradients.
Stochastic gradient descent on l1 loss converges to true parameter in online robust regression.
problem Online robust regression with adversarial noise.
method Stochastic gradient descent on l1 loss.
result Converges to true parameter at a rate independent of contaminated measurements.
Study on bias and extrapolation in LSA with Markovian data, showing bias reduction with Richardson-Romberg extrapolation.
problem Bias in LSA with constant stepsizes and Markovian data.
method Viewing LSA as a Markov chain, proving convergence and bias expansion, and applying Richardson-Romberg extrapolation.
result Bias is proportional to the stepsize up to higher order terms, and Richardson-Romberg extrapolation reduces the bias.
Paper improves confidence set construction for SGD using multiplier bootstrap.
problem Constructing accurate confidence sets for SGD.
method Multiplier bootstrap procedure for non-asymptotic validity.
result Derives approximation rates up to 1 / n 1/\sqrt{n} 1/ n for convex distance. Paper analyzes convergence of two time-scale stochastic approximation using martingale approach.
problem Analyzing convergence of two time-scale stochastic approximation algorithms.
method Uses martingale approach to establish convergence conditions and rates.
result Establishes different rates of convergence for fast and slow subsystems.
The paper analyzes the performance of constant step-size stochastic approximation algorithms.
problem Approximating solutions to root finding problems in optimization and machine learning.
method Examines stochastic approximation algorithms with constant step-size, proving convergence and analyzing the limiting behavior of averaged estimates.
result The Polyak-Ruppert-style averaged estimates converge to the true solution with optimal covariance, providing insights for practitioners.
Tail averaging consists in averaging the last examples in a stream. Common techniques either have a memory requirement which grows with the number of samples to average, are not available at every timestep or do not accomodate growing windows. We propose two techniques with a low constant memory cost that perform tail …
New RL algorithms improve average-reward performance.
problem Improving RL algorithms for average-reward criteria.
method Developed novel algorithms addressing average-reward criterion directly.
result ATRPO significantly outperforms TRPO in challenging MuJuCo environments.