Cyclical learning rate improves neural machine translation performance.
problem Optimizing learning rate for neural machine translation.
method Applied cyclical learning rate to transformer-based neural networks.
result Cyclical learning rate significantly impacts neural machine translation performance.
Policy gradient algorithm with variable learning rates achieves near-optimal performance in multi-arm bandit problems.
problem Optimizing a policy gradient algorithm for multi-arm bandit problems with variable learning rates.
method Applied Foster-Lyapunov techniques to analyze a Markov chain formed by the state of the algorithm.
result The policy gradient algorithm converges to the optimal arm with logarithmic or poly-logarithmic regret.
LALR adapts learning rate for faster convergence in regression and neural nets.
problem Finding optimal learning rates for faster convergence in regression and neural networks.
method Lipschitz continuity theory applied to Mean Absolute Error and Quantile loss functions.
result Adaptive learning rate policy enables up to 20x faster convergence.
Learning Rate (LR) is an important hyper-parameter to tune for effective training of deep neural networks (DNNs). Even for the baseline of a constant learning rate, it is non-trivial to choose a good constant value for training a DNN. Dynamic learning rates involve multi-step tuning of LR values at various stages of th…
New guarantees for ERM with adaptively collected data.
problem Failure of ERM guarantees with adaptively collected data.
method Importance sampling weighted ERM algorithm with maximal inequality.
result First generalization guarantees and fast convergence rates for adaptively collected data.
Minimal assumptions analysis of Q-learning with time-varying policies.
problem Finite-time analysis of Q-learning with time-varying policies for discounted MDPs.
method Minimal assumptions, Poisson equation decomposition, sensitivity analysis.
result Established convergence rate and sample complexity for Q-learning.
New methods estimate policy value and gradients for deterministic policies from off-policy data.
problem Estimating policy value and gradients for deterministic policies from off-policy data.
method Proposed new doubly robust estimators based on kernelization approaches.
result Demonstrated a rate independent of horizon length for policy value and gradient estimation.
Successful deployment of machine learning algorithms in healthcare requires careful assessments of their performance and safety. To date, the FDA approves locked algorithms prior to marketing and requires future updates to undergo separate premarket reviews. However, this negates a key feature of machine learning--the …
Study on policy gradient for stochastic bandits using diffusion approximation.
problem Improving policy gradient methods for stochastic bandits with optimal regret bounds.
method Continuous-time diffusion approximation of policy gradient with learning rate analysis.
result Proved optimal regret bound of O ( k log ( k ) log ( n ) / η ) O(k \log(k) \log(n) / η) O ( k log ( k ) log ( n ) / η ) for η = O ( Δ 2 / log ( n ) ) η= O(Δ^2/\log(n)) η = O ( Δ 2 / log ( n )) . Robotic table tennis learns efficient policies to return balls at 100Hz.
problem Developing efficient robotic table tennis strategies.
method Model-free reinforcement learning using evolutionary search on CNN-based policies.
result Robots can develop multi-modal styles (forehand and backhand) with 80% return rate.
Develops DPG methods for continuous-time RL with deterministic policies.
problem High variance and slow convergence in stochastic policy RL methods.
method Derives continuous-time policy gradient formula and proposes CT-DDPG algorithm.
result CT-DDPG achieves superior stability and faster convergence in continuous-time RL.
Entropy regularized algorithms such as Soft Q-learning and Soft Actor-Critic, recently showed state-of-the-art performance on a number of challenging reinforcement learning (RL) tasks. The regularized formulation modifies the standard RL objective and thus generally converges to a policy different from the optimal gree…
Counterfactual policy evaluation improves autonomous driving policies' generalization.
problem Learnt policies often fail to generalize and handle novel situations.
method Introduces counterfactual policy evaluation using counterfactual worlds.
result Significantly decreases collision-rate while maintaining high success-rate.
A drone catches another agile drone using competitive reinforcement learning.
problem Intercepting an agile drone with another agile drone.
method Formulated as a Competitive Reinforcement Learning problem, trained with PPO, using a high-fidelity simulation environment.
result Trained policies outperform common heuristic baselines in catch rate, time to catch, and crash rate.
New reinforcement learning algorithm achieves instance-optimal sample complexity.
problem Achieving low regret and identifying optimal policies in reinforcement learning.
method A novel planning-based algorithm that explicitly accounts for state visitation distributions.
result The proposed algorithm attains nearly minimax optimal sample complexity, improving over worst-case bounds.
Linear Q-learning converges to a bounded set without divergence.
problem Proving linear Q-learning does not diverge and converges to a bounded set.
method No modifications to the original linear Q-learning algorithm, no Bellman completeness or near-optimality assumptions, only an ε-softmax behavior policy with adaptive temperature.
result First L 2 L^2 L 2 convergence rate of linear Q-learning iterates to a bounded set. This paper tackles robust policy learning under concept drifts, improving upon existing methods.
problem Tackles robust policy learning under concept drifts, improving upon existing methods.
method Develops a doubly-robust estimator and a learning algorithm to maximize policy value within a given policy class.
result The proposed algorithm achieves sub-optimality gap of the order κ ( Π ) n − 1 / 2 κ(Π)n^{-1/2} κ ( Π ) n − 1/2 , demonstrating substantial improvement over existing benchmarks. Trust region policy optimization (TRPO) is a popular and empirically successful policy search algorithm in Reinforcement Learning (RL) in which a surrogate problem, that restricts consecutive policies to be 'close' to one another, is iteratively solved. Nevertheless, TRPO has been considered a heuristic algorithm inspi…
Single-timescale actor-critic finds globally optimal policy.
problem Finding globally optimal policy in reinforcement learning.
method Simultaneous actor and critic updates with linear or deep neural network approximations.
result Actor sequence converges to globally optimal policy at O ( K − 1 / 2 ) O(K^{-1/2}) O ( K − 1/2 ) rate. The paper tackles adaptive policy selection to maximize social welfare, achieving optimal regret bounds.
problem Maximizing social welfare through adaptive policy selection, considering both private utility and public revenue.
method The approach involves learning response functions through experimentation, deriving lower and upper bounds for regret, and using algorithms like Exp3.
result The algorithm achieves optimal regret bounds, showing that welfare maximization is harder than multi-armed bandit problems.
Tabular Q-learning outperforms advanced RL methods in monetary policy.
problem Dynamic setting of short-term interest rates to stabilize inflation and unemployment under uncertain macroeconomic conditions.
method Discrete-action Markov Decision Process with tabular Q-learning, SARSA, Actor-Critic, Deep Q-Networks, Bayesian Q-learning, POMDP formulations.
result Standard tabular Q-learning achieved the best performance (-615.13 +- 309.58 mean return) compared to advanced RL methods and traditional policy rules.
Paper achieves ε − 2 ε^{-2} ε − 2 sample complexity for actor-critic methods with minimal assumptions.
problem Achieving ε − 2 ε^{-2} ε − 2 sample complexity for actor-critic methods under minimal assumptions. method Single-loop, single-timescale implementation; coupled Lyapunov drift framework.
result First i l d e O ( ε − 2 ) ilde{\mathcal{O}}(ε^{-2}) i l d e O ( ε − 2 ) sample complexity guarantee for finding an ε ε ε -optimal policy. New method estimates and optimizes policy differences using orthogonal learning.
problem Offline reinforcement learning with safety concerns and cost limitations.
method Dynamic R-learner for estimating and optimizing Q π ( s , 1 ) − Q π ( s , 0 ) Q^π(s,1)-Q^π(s,0) Q π ( s , 1 ) − Q π ( s , 0 ) , leveraging orthogonal estimation. result Consistent policy optimization with improved convergence rates.
Unified framework for stable RL learning with theoretical guarantees.
problem Lack of systematic theoretical principles guiding RL post-training methods.
method Unified theoretical framework for policy-gradient estimators and optimization algorithms.
result Establishes unbiasedness, variance expressions, and convergence guarantees.
Softmax policy gradient methods converge at O ( 1 / t ) O(1/t) O ( 1/ t ) rate with constants depending on problem and initialization.
problem Understanding convergence rates of softmax policy gradient methods in tabular settings.
method Analysis of softmax policy gradient and entropy regularized policy gradient methods, using Łojasiewicz inequality and lower bounds.
result Entropy regularization improves convergence rate from O ( 1 / t ) O(1/t) O ( 1/ t ) to O ( e − c ⋅ t ) O(e^{-c \cdot t}) O ( e − c ⋅ t ) . The paper tackles robust policy learning in MDPs using statistical methods.
problem Offline data-driven sequential decision making in MDPs.
method Evaluates policies using average rewards centered at policy-induced stationary distributions. Developed a statistically efficient method for estimating robust optimal policies.
result Established a rate-optimal regret bound up to a logarithmic factor.
Adaptive OMD reduces variance in learning optimal strategies for imperfect information games.
problem High variance in learning optimal strategies for imperfect information games.
method Fixed sampling approach with locally applied Online Mirror Descent (OMD) algorithm.
result Convergence rate of i l d e O ( T − 1 / 2 ) ilde{\mathcal{O}}(T^{-1/2}) i l d e O ( T − 1/2 ) with high probability. Unified framework for risk-aware policy learning in contextual bandits.
problem Optimizing decision rules in high-stakes domains with adverse outcomes.
method Distributional framework for Lipschitz-continuous risk functionals, with novel empirical concentration inequalities.
result Data-dependent suboptimality bounds with an i l d e O ( 1 / n ) ilde{\mathcal{O}}(1/\sqrt{n}) i l d e O ( 1/ n ) rate, matching risk-neutral offline policy optimization. Rate change calculations in the literature involve deterministic methods that measure the change in premium for a given policy. The definition of rate change as a statistical parameter is proposed to address the stochastic nature of the premium charged for a policy. It promotes the idea that rate change is a property o…
Large deviations theory applied to policy gradient methods.
problem Understanding convergence of policy gradient methods in reinforcement learning.
method Large deviation rate function and contraction principle from large deviations theory.
result Convergence properties of policy gradient methods can be extended to various policy parametrizations.
The paper analyzes reinforcement learning methods for estimating weights and quality functions with fast convergence rates.
problem Estimating weights and quality functions in reinforcement learning with function approximation.
method The paper uses minimax methods for estimating marginal importance weights and q-functions.
result The minimax approach enables fast rates of convergence for weights and quality functions, achieving first-order efficiency.
Sample efficiency is critical in solving real-world reinforcement learning problems, where agent-environment interactions can be costly. Imitation learning from expert advice has proved to be an effective strategy for reducing the number of interactions required to train a policy. Online imitation learning, which inter…
The paper tackles robust policy learning from multiple data sources.
problem Learning a policy that generalizes across diverse settings from multiple heterogeneous data sources.
method Proposes a minimax regret optimization objective and a policy learning algorithm combining doubly robust offline policy evaluation and no-regret learning.
result Achieves minimal worst-case mixture regret up to a moderated vanishing rate of the total data across all sources.
Parrot learns optimal cache replacement policies using imitation learning.
problem Improving cache hit rates in complex access patterns.
method Imitation learning approach using Belady's oracle policy.
result Parrot increases cache hit rates by 61% on a web search benchmark.
Study introduces a new framework for policy learning without positivity assumption.
problem Learning optimal treatment assignment policies from observational data with constraints.
method Incremental propensity score policies and semiparametric efficiency theory.
result Validated framework's performance through numerical experiments.
This paper investigates to what extent one can improve reinforcement learning algorithms. Our study is split in three parts. First, our analysis shows that the classical asymptotic convergence rate O ( 1 / N ) O(1/\sqrt{N}) O ( 1/ N ) is pessimistic and can be replaced by O ( ( log ( N ) / N ) β ) O((\log(N)/N)^β) O (( log ( N ) / N ) β ) with 1 2 ≤ β ≤ 1 \frac{1}{2}\leq β\leq 1 2 1 ≤ β ≤ 1 and N N N the number…
The paper develops a method to learn cost-optimal sequential testing policies from retrospective data.
problem Learning cost-optimal sequential decision policies from retrospective data with missing test results.
method Doubly robust Q-learning framework with path-specific inverse probability weights.
result The method reduces testing cost without compromising predictive accuracy.
Study minimax rates for online learning with time-varying dynamics.
problem Online learning with time-varying state and cost dynamics.
method Non-constructive upper and lower bounds, complexity and stability terms.
result Characterization of minimax rates and necessary conditions for learnability.
Proposes DR algorithms for distributionally robust off-policy evaluation and learning.
problem Sensitive to environment distribution shifts in offline observational data.
method Doubly robust and distributionally robust approaches for OPE/L.
result Achieves semiparametric efficiency and fast regret rate.
We decode latent states in Block MDPs and learn near-optimal policies.
problem Model estimation and reward-free learning in Block MDPs.
method Information-theoretical lower bound and efficient model estimation algorithm.
result Our algorithm approaches the information-theoretical limit for latent state decoding and converges to optimal policies.
Study uses ML to analyze how interest rates affect fund returns, finding gradient boosting is effective.
problem Understanding how interest rate changes impact fund returns.
method Combines Machine Learning and causal inference, using Double Machine Learning framework.
result Gradient boosting is useful for predicting fund returns, showing a significant negative effect of interest rate increases.
Develops a new reinforcement learning framework for complex control problems.
problem Continuous-time extended mean field control with deterministic policies.
method Model-free sensitivity formula, deterministic policy gradient, local value and advantage-rate representations.
result Demonstrates efficiency, stability, and robustness in solving complex control problems.
New method for RL with general utilities using variational policy gradient.
problem Optimizing policies with general concave utility functions in RL.
method Derives Variational Policy Gradient Theorem, develops variational Monte Carlo gradient estimation algorithm.
result Global convergence to optimal policy for general objectives, exponential convergence under strong convexity.
The paper proposes an interpretable off-policy learning algorithm for medical treatments.
problem Lack of interpretable methods for personalized treatment decisions from observational data.
method Hyperbox search approach for interpretable policies in disjunctive normal form.
result The proposed algorithm outperforms state-of-the-art methods in terms of regret and is rated highly interpretable by clinical experts.
Policy gradient methods achieve linear convergence in simple MDPs.
problem Analyzing convergence rates of policy gradient methods in finite MDPs.
method Connections with policy iteration to show linear convergence with large step-sizes.
result Policy gradient methods succeed with large step-sizes and achieve linear rate of convergence.
Study Federated RL with diverse constraints, proposing new optimization methods.
problem Solving reinforcement learning with multiple constraints in federated learning.
method Federated primal-dual policy optimization methods based on policy gradient methods.
result FedNPG achieves global convergence with an i l d e O ( 1 / T ) ilde{O}(1/\sqrt{T}) i l d e O ( 1/ T ) rate. New approach uses PDE learning for faster RL fine-tuning.
problem Learning optimal control policy for diffusion process.
method Solves variational inequality based on HJB equations.
result Shows fine-tuning can be done via supervised regression.
A new method for learning policies in multiple environments.
problem Learning policies that work in different but related environments.
method Decentralized entropy-regularized policy gradient method.
result The method can learn effective policies in multiple environments.