The paper analyzes deep neural networks using control theory to set a time limit for their convergence.
problem Understanding the finite-time convergence of deep neural networks.
method Lyapunov based analysis of the loss function, control theory framework, finite-time control of non-linear systems.
result A priori guarantees of finite-time convergence for deep neural networks are provided.
Study on SA with heavy-tailed and LRD noise, establishing finite-time bounds.
problem Analyzing stochastic approximation under heavy-tailed and LRD noise.
method Noise-averaging argument to regularize impact of non-classical noise.
result Established first finite-time moment bounds for SA under heavy-tailed and LRD noise.
This work analyzes actor-critic methods for faster convergence.
problem Finite-time analysis and sample complexity of two-time-scale actor-critic methods.
method Non-asymptotic analysis under non-i.i.d. setting, proving convergence to first-order stationary point.
result Actor-critic method finds a first-order stationary point with ildeO(ε−2.5) sample complexity. New analysis of stochastic approximation with non-expansive mappings.
problem Finite-time analysis of two-time-scale stochastic approximation with non-expansive mappings.
method Studied two-time-scale stochastic approximation algorithms with non-expansive mappings and projection steps.
result Last-iterate mean square residual error decays at a rate O(1/k1/4−ε). Paper analyzes finite-time convergence of double Q-learning.
problem Overestimation issue in Q-learning.
method Finite-time analysis of double Q-learning.
result Convergence to ε-accurate neighborhood in finite iterations.
Q-learning with neural network function approximation (neural Q-learning for short) is among the most prevalent deep reinforcement learning algorithms. Despite its empirical success, the non-asymptotic convergence rate of neural Q-learning remains virtually unknown. In this paper, we present a finite-time analysis of a…
Estimates for VPMCF show ancient MCF solutions and finite-time behavior.
problem Volume Preserving Mean Curvature Flow (VPMCF) behavior and singularities.
method Nonlocal estimates and blowup analysis.
result Ancient solutions to MCF and finite-time behavior of VPMCF.
Improved TD learning with tail averaging and regularization achieves optimal convergence rates.
problem Convergence analysis of TD learning with linear function approximation.
method Tail-averaging and regularization applied to TD learning algorithm.
result Achieves optimal O(1/t) convergence rate in expectation and with high probability. Paper analyzes convergence of dynamic policy gradient for MDPs, improving performance in finite-time problems.
problem Optimal policies in finite-time MDPs are not stationary and require epoch-specific training.
method Introduces dynamic policy gradient combining dynamic programming and policy gradient, analyzes convergence for softmax parametrisation.
result Dynamic policy gradient training exploits finite-time structure, leading to better convergence bounds.
Logarithmic regret achieved in continuous-time linear-quadratic reinforcement learning.
problem Optimizing control actions in unknown continuous-time systems over a finite time horizon.
method Least-squares algorithm based on continuous-time observations and controls, with perturbation analysis and parameter estimation error analysis.
result Logarithmic regret bound of order O((lnM)(lnlnM)). We investigate finite-time decoupled convergence in nonlinear two-time-scale stochastic approximation.
problem Achieving decoupled convergence in nonlinear two-time-scale stochastic approximation.
method Nested local linearity assumption, suitable step size selection, convergence analysis of matrix cross term, fourth-order moment convergence rates.
result Finite-time decoupled convergence rates can be achieved in nonlinear two-time-scale stochastic approximation with proper step size selection.
I introduce and analyse an anytime version of the Optimally Confident UCB (OCUCB) algorithm designed for minimising the cumulative regret in finite-armed stochastic bandits with subgaussian noise. The new algorithm is simple, intuitive (in hindsight) and comes with the strongest finite-time regret guarantees for a hori…
The paper analyzes finite-time singularities in Spin(7)-structure flows using Shi-type estimates.
problem Analyzing finite-time singularities in Spin(7)-structure flows.
method Proves Shi-type derivative estimates and shows that Λ(x,t) must blow up at finite-time singularities.
result Establishes a general analytic framework for studying Spin(7)-structure flows.
The question of the optimality of Thompson Sampling for solving the stochastic multi-armed bandit problem had been open since 1933. In this paper we answer it positively for the case of Bernoulli rewards by providing the first finite-time analysis that matches the asymptotic rate given in the Lai and Robbins lower boun…
We study finite sample properties of estimators of power-law cross-correlations -- detrended cross-correlation analysis (DCCA), height cross-correlation analysis (HXA) and detrending moving-average cross-correlation analysis (DMCA) -- with a special focus on short-term memory bias as well as power-law coherency. Presen…
I analyse the frequentist regret of the famous Gittins index strategy for multi-armed bandits with Gaussian noise and a finite horizon. Remarkably it turns out that this approach leads to finite-time regret guarantees comparable to those available for the popular UCB algorithm. Along the way I derive finite-time bounds…
Linear two-timescale stochastic approximation (SA) scheme is an important class of algorithms which has become popular in reinforcement learning (RL), particularly for the policy evaluation problem. Recently, a number of works have been devoted to establishing the finite time analysis of the scheme, especially under th…
First-order method solves stochastic bilevel optimization with linear constraints.
problem Stochastic bilevel optimization with linear constraints and noise.
method Developed a novel framework using gradient-based techniques and smoothed penalty functions.
result Achieved finite-time convergence guarantees for (δ,ε)-Goldstein stationary points. The OLS estimator optimally identifies stable linear systems with a finite number of samples.
problem Identifying stable linear systems with a finite number of samples.
method Finite-time analysis of the Ordinary Least Squares (OLS) estimator for stable linear systems.
result The OLS estimator achieves optimal sample complexity for stable systems, matching existing lower bounds up to universal factors.
Novel analysis of EFP for finite-sum problems in neural networks.
problem Optimization of two-layer neural networks in the mean-field regime.
method Primal-dual analysis of entropic fictitious play (EFP) for finite-sum problems.
result Established global convergence guarantees for EFP dynamics.
Constructing eigenfunctions for finite-time singularities in Lagrangian mean curvature flow
problem Constructing eigenfunctions for finite-time singularities in Lagrangian mean curvature flow
method Constructing eigenfunctions for finite-time singularities in Lagrangian mean curvature flow
result Identifying the lowest eigenfunction with the scaling mode of the special Lagrangian desingularization
Minimal assumptions analysis of Q-learning with time-varying policies.
problem Finite-time analysis of Q-learning with time-varying policies for discounted MDPs.
method Minimal assumptions, Poisson equation decomposition, sensitivity analysis.
result Established convergence rate and sample complexity for Q-learning.
New analysis shows actor-critic method converges efficiently in practical settings.
problem Understanding finite-time convergence of single-timescale actor-critic methods.
method Investigated online single-timescale actor-critic algorithm with linear function approximation and Markovian sampling.
result Proved convergence to ε-approximate stationary point with sample complexity of O(ε^(-2)).
Paper analyzes finite-time guarantees for preference-based RL.
problem Understanding finite-time guarantees for preference-based RL.
method Combines dueling bandits and policy search to navigate state space.
result Identifies best policy up to accuracy ε with high probability.
Paper analyzes SVGD algorithm for non-asymptotic convergence.
problem Optimizing a set of particles to approximate a target probability distribution.
method Finite time analysis of SVGD algorithm, providing descent lemma and convergence rates.
result SVGD algorithm decreases the objective at each iteration and converges to the target distribution.
In this paper, we analyze the finite sample complexity of stochastic system identification using modern tools from machine learning and statistics. An unknown discrete-time linear system evolves over time under Gaussian noise without external inputs. The objective is to recover the system parameters as well as the Kalm…
We tackle the problem of online reward maximisation over a large finite set of actions described by their contexts. We focus on the case when the number of actions is too big to sample all of them even once. However we assume that we have access to the similarities between actions' contexts and that the expected reward…
New bounds derived for KG algorithm's performance in finite time.
problem Best arm identification problem in multi-armed bandit.
method Theoretical analysis of finite-time performance, deriving bounds for sample allocation, error probability, and regret.
result Upper and lower bounds for the probability of error and simple regret of the KG algorithm.
Temporal difference learning (TD) is a simple iterative algorithm used to estimate the value function corresponding to a given policy in a Markov decision process. Although TD is one of the most widely used algorithms in reinforcement learning, its theoretical analysis has proved challenging and few guarantees on its s…
Paper analyzes NAC with neural networks for efficient policy optimization.
problem Improving sample and iteration complexity in policy optimization.
method Entropy regularization, averaging, neural network approximation, and optimization techniques.
result Entropy regularization and averaging ensure stability and sharp sample complexity bounds.
Bitcoin volatility analysis shows decreasing HE with longer sampling periods.
problem Understanding volatility patterns in Bitcoin using different sample sizes.
method Examined Bitcoin data to analyze Hurst exponent (HE) and multifractality.
result HE decreases as sampling period increases, indicating rough volatility.
Theoretical justification for asymmetric actor-critic algorithms in reinforcement learning.
problem Lack of precise theoretical justification for asymmetric actor-critic algorithms in reinforcement learning.
method Adapting a finite-time convergence analysis to the asymmetric actor-critic setting with linear function approximators.
result A finite-time bound reveals that the asymmetric critic eliminates aliasing errors in the agent state.
Diffusion approximation provides weak approximation for stochastic gradient descent algorithms in a finite time horizon. In this paper, we introduce new tools motivated by the backward error analysis of numerical stochastic differential equations into the theoretical framework of diffusion approximation, extending the …
Paper analyzes PSGLD for adaptive IRL with finite-sample bounds.
problem Estimating cost function of a forward learner using noisy gradients.
method Passive stochastic gradient Langevin dynamics (PSGLD) algorithm.
result Explicit bounds on 2-Wasserstein distance between PSGLD sample measure and stationary measure.
Uniform-in-time analysis for Stein Variational Gradient Descent across various metrics.
problem Understanding long-term behavior of finite-particle systems in relation to their mean-field limits.
method Developed uniform-in-time propagation-of-chaos results for continuous-time SVGD using cutoff strategies and finite-dimensional theories.
result Uniform-in-time propagation-of-chaos bounds in various metrics, including Langevin kernel Stein discrepancy, Wasserstein-1, and Wasserstein-2 distances.
Despite the increasing interest in multi-agent reinforcement learning (MARL) in multiple communities, understanding its theoretical foundation has long been recognized as a challenging problem. In this work, we address this problem by providing a finite-sample analysis for decentralized batch MARL with networked agents…
First, we consider the problem of hedging in complete binomial models. Using the discrete-time Föllmer-Schweizer decomposition, we demonstrate the equivalence of the backward induction and sequential regression approaches. Second, in incomplete trinomial models, we examine the extension of the sequential regression app…
The paper provides bounds on estimation error in a distributed online learning setting.
problem Estimating an unknown parameter in a distributed and online manner with finite sample guarantees.
method Proposes a distributed online estimation algorithm that improves accuracy through communication, providing non-asymptotic bounds on estimation error.
result Demonstrates a trade-off between estimation error and communication costs, and determines a stopping time for communication based on desired accuracy.
We consider the dynamics of a linear stochastic approximation algorithm driven by Markovian noise, and derive finite-time bounds on the moments of the error, i.e., deviation of the output of the algorithm from the equilibrium point of an associated ordinary differential equation (ODE). We obtain finite-time bounds on t…
The paper offers precise bounds for averaged LSA iterates in linear systems.
problem Computing approximate solutions of linear systems with noisy observations.
method Finite-time analysis of LSA algorithms with Polyak-Ruppert averaging.
result Sharp high-probability bounds for averaged LSA iterates.
This work analyzes Q-learning with adaptive stepsizes for finite-time convergence.
problem Finite-time convergence analysis for average-reward Q-learning with adaptive stepsizes. method Adaptive stepsizes as local clocks, time-inhomogeneous Markovian reformulation, almost-sure time-varying bounds, conditioning arguments, and Markov chain concentration inequalities.
result Convergence rates of ildeO(1/k) for mean-square and pointwise mean-square convergence. SARSA is an on-policy algorithm to learn a Markov decision process policy in reinforcement learning. We investigate the SARSA algorithm with linear function approximation under the non-i.i.d.\ data, where a single sample trajectory is available. With a Lipschitz continuous policy improvement operator that is smooth eno…
We study an extension of the classic stochastic multi-armed bandit problem which involves multiple plays and Markovian rewards in the rested bandits setting. In order to tackle this problem we consider an adaptive allocation rule which at each stage combines the information from the sample means of all the arms, with t…
Study on eigenvalue distribution of correlated time series, showing deformation of Marchenko-Pastur distribution.
problem Eigenvalue distribution of Wishart matrix with temporal correlation.
method Analysis of moments and convergence to deformed Marchenko-Pastur distribution for Gaussian process with temporal correlation.
result Eigenvalue distribution converges to deformed Marchenko-Pastur distribution with longer tail and higher peak.
New Thompson sampling algorithm reduces regret for exponential family bandits.
problem Minimizing regret in multi-armed bandit problems with exponential family rewards.
method Proposes ExpTS and ExpTS+ algorithms using novel sampling distributions. result Minimizes both finite-time and asymptotic regret for exponential family rewards.
This study analyzes LTS in sparse models with finite sample error bounds.
problem Robust regression in high-dimensional sparse models with limited data.
method Non-asymptotic analysis of LTS error bounds.
result Established finite sample error bounds for LTS in sparse models.
Proposes a TS approach for Bayesian optimization with preferential feedback.
problem Optimizing with preference feedback in complex applications.
method Uses Thompson Sampling with a dueling kernel and anchor invariance.
result Performance matches standard TS for scalar feedback in finite time.
New bounds for SA with arbitrary norm contractions and Markovian noise.
problem Finite-time analysis of two-time-scale stochastic approximation with arbitrary norm contractions and Markovian noise.
method Use of generalized Moreau envelope for arbitrary norm contractions and solutions of Poisson equation for Markovian noise.
result Mean square error decays at rates of O(1/n2/3) and O(1/n) under different conditions.