Paper addresses offline policy evaluation in RL, achieving near-optimal bounds for various policy classes.
problem Evaluate all policies in a class simultaneously for offline RL.
method Uniform convergence in OPE for various policy classes, achieving optimal episode complexity.
result Achieves optimal episode complexity of O(H^3/d_mε^2) for identifying ε-optimal policies.
UCPO improves diversity in reinforcement learning models, maintaining high accuracy.
problem RLVR objectives often lead to diversity collapse, reducing coverage of correct solutions.
method UCPO adds a conditional uniformity penalty to GRPO, redistributing probability mass.
result UCPO improves Pass@K and diversity while maintaining competitive Pass@1 accuracy.
New algorithm learns policies without uniform overlap assumption.
problem Learning optimal policies from non-uniformly collected data.
method Pessimistic Policy Learning (PPL) using lower confidence bounds.
result Efficient policy learning for adaptively collected data.
New offline RL method handles average-reward MDPs with single-policy coverage.
problem Challenges in offline reinforcement learning due to distribution shift and non-uniform coverage.
method Develops an algorithm based on pessimistic discounted value iteration with quantile clipping.
result First fully single-policy sample complexity bound for average-reward offline RL.
New framework improves restless bandit policies for large numbers of arms.
problem Efficiently compute policies for large numbers of arms in restless bandit problems.
method Follow-the-Virtual-Advice framework, converting single-armed policies to N-armed policies.
result Achieves an O(1/\sqrt{N}) optimality gap in both discrete and continuous settings.
New algorithm FLUTE achieves uniform-PAC convergence in RL with linear approx.
problem RL with linear function approximation lacks uniform-PAC guarantees.
method FLUTE algorithm with minimax value function estimator and multi-level partition scheme.
result Uniform-PAC convergence to optimal policy with high probability.
A streamlined DRL algorithm improves sample efficiency without entropy maximization.
problem Improving sample efficiency in off-policy DRL algorithms.
method Output normalization and non-uniform sampling.
result Proposed algorithm matches SAC's performance without entropy maximization and improves sample efficiency.
Paper introduces risk assessment for contextual bandits without experiments.
problem Evaluate policies using logged data in context bandits.
method Lipschitz risk functionals and Off-Policy Risk Assessment (OPRA) framework.
result OPRA provides finite sample guarantees for various risk estimates.
Paper tackles robust policy learning with incomplete data.
problem Learning policies from past data assumes future environment is the same, which is often false.
method Develops a distributionally robust policy evaluation and learning algorithm.
result Proposed algorithm provides robustness to adversarial perturbations and covariate shifts.
First-best climate policy is a uniform carbon tax which gradually rises over time. Civil servants have complicated climate policy to expand bureaucracies, politicians to create rents. Environmentalists have exaggerated climate change to gain influence, other activists have joined the climate bandwagon. Opponents to cli…
This study optimizes offline reinforcement learning methods for various tasks without rewards.
problem Optimizing offline reinforcement learning for multiple tasks without rewards.
method Designing a new model-based approach with singleton absorbing MDPs to achieve optimal convergence rates.
result Achieved optimal convergence rates for offline reinforcement learning in various settings.
Paper achieves ε − 2 ε^{-2} ε − 2 sample complexity for actor-critic methods with minimal assumptions.
problem Achieving ε − 2 ε^{-2} ε − 2 sample complexity for actor-critic methods under minimal assumptions. method Single-loop, single-timescale implementation; coupled Lyapunov drift framework.
result First i l d e O ( ε − 2 ) ilde{\mathcal{O}}(ε^{-2}) i l d e O ( ε − 2 ) sample complexity guarantee for finding an ε ε ε -optimal policy. This paper introduces a new metric, ULI, for RL that ensures both cumulative and instantaneous performance.
problem High-stakes applications require RL algorithms to avoid playing bad policies.
method Introduces uniform last-iterate (ULI) guarantee, a stronger metric capturing both cumulative and instantaneous performance.
result ULI directly implies near-optimal cumulative performance across various metrics, but not the other way around.
Pessimistic Minimax Value Iteration finds efficient NE policies from offline data.
problem Finding an approximate Nash equilibrium in offline Markov games with non-uniform coverage.
method Pessimistic Minimax Value Iteration (PMVI) constructs pessimistic value function estimates and solves NEs.
result Established a nearly minimax optimal result for offline Markov games with function approximation.
Study uses contextual bandits to optimize charity exposure in donation solicitation.
problem Optimizing charity exposure in donation solicitation using survey responses.
method Adaptive experiment design to balance cumulative regret minimization and simple regret minimization.
result Adaptive experimentation yields better policy learning outcomes than uniform randomization.
Study uses machine learning to estimate effective policies in settings with hidden individual actions.
problem Estimating effective policies in settings with hidden individual actions.
method Instrumental Regression and Generalized Method of Moments (GMM) estimator.
result Demonstrates how to estimate a good contract in principal-agent problems.
A new method for efficient exploration in reinforcement learning.
problem Improving exploration in reinforcement learning agents.
method State Marginal Matching (SMM) to learn policies matching a target state distribution.
result Agents that optimize SMM explore faster and adapt quicker to new tasks.
VRER selectively reuses samples to improve policy optimization in complex systems.
problem Lack of effective reuse of historical samples in reinforcement learning.
method Variance reduction based experience replay (VRER) framework.
result VRER accelerates policy optimization and enhances performance.
New method improves solving combinatorial optimization problems with smoothed policies.
problem Solving combinatorial optimization problems repeatedly with varying instances.
method Smoothed policies with controlled random perturbations to linear oracle, leading to differentiable surrogate risk.
result Generalization bound decomposes excess risk into bias, estimation, and optimization components.
New algorithm learns optimal policy for average reward MDPs with sample complexity matching lower bound.
problem Learning optimal policy for average reward in uniformly ergodic MDPs.
method Developed an estimator with sample complexity of O(|S||A|t_{mix}ε^{-2}).
result First algorithm to match lower bound of existing literature.
Improved deep learning model deployment on tiny MCUs with mixed-precision quantization.
problem Memory limitations prevent accurate deployment of DNN models on tiny MCUs.
method Automated mixed-precision quantization using Reinforcement Learning for MCU constraints.
result Mixed-precision models achieve high accuracy with uniform quantization policies.
Paper formulates mutual information optimal control for discrete-time systems.
problem Optimal control of discrete-time linear systems with mutual information.
method Formulates MIOCP as an extension of MEOCP, derives optimal policy and prior, proposes alternating minimization algorithm.
result Proposes an alternating minimization algorithm for MIOCP.
Consider the problem of a controller sampling sequentially from a finite number of N ≥ 2 N \geq 2 N ≥ 2 populations, specified by random variables X k i X^i_k X k i , i = 1 , … , N , i = 1,\ldots , N, i = 1 , … , N , and k = 1 , 2 , … k = 1, 2, \ldots k = 1 , 2 , … ; where X k i X^i_k X k i denotes the outcome from population i i i the k t h k^{th} k t h time it is sampled. It is assumed that for each fixed i i i , $\{…
This work improves policy optimization by maximizing entropy of state distribution, leading to better exploration.
problem Lack of exploration in state space when maximizing policy entropy.
method Proposes maximizing the entropy of a lower bound approximation to the state weighting distribution, based on latent space representation.
result Entropy regularization based on marginal state distribution achieves superior state space coverage and better performance in various domains.
New adaptive temperature selection improves parallel tempering efficiency.
problem Enhancing mixing in multi-modal distributions using parallel tempering.
method Adaptive temperature selection using policy gradient approach.
result Lower integrated autocorrelation times achieved compared to traditional methods.
Wasserstein Policy Learning for Distributional Outcomes
problem Offline policy learning with distribution-valued outcomes
method Establishing statistical guarantees for policy learning framework
result Proven leading dependence on N and N-dim(Π) for finite-sample regret
Unified stopping rules ensure accurate policies in contextual learning.
problem Stopping data collection to ensure accurate policies in personalized decision problems.
method Developed unified stopping rules based on GLR statistics for pairwise action comparisons.
result Unified stopping rules achieve target precision with fewer samples than benchmarks.
New methods tackle robust reinforcement learning in sparse, corrupted data.
problem Tackles robust reinforcement learning in sparse, corrupted data.
method Proposes actor-critic methods with sparse robust estimator oracles.
result First non-vacuous guarantees in high-dimensional sparse MDPs with single-policy concentrability coverage.
New estimator reduces risk in slate bandits by leveraging Bayes risk criterion.
problem Evaluating slate policies using logged data when policies factorize over slots.
method Developed a new estimator using a control variate approach, showing risk improvement over existing methods.
result The new estimator has lower risk than the pseudoinverse estimator in slate bandit problems.
Paper proposes a new approach for agents to explore environments efficiently.
problem Learning policies that explore uniformly and mix quickly in environments without external rewards.
method Introduces a surrogate objective to maximize entropy and develops a model-based reinforcement learning algorithm, IDE 3 ^{3} 3 AL. result Demonstrates improved exploration and mixing in hard-exploration tasks.
This paper tackles post-trade allocation inefficiencies and presents a uniform return allocation method.
problem Return divergence among accounts after trade allocation.
method Systematic treatment of trade allocation risk, presenting a uniform return allocation method.
result Uniform allocation of returns irrespective of the number of accounts and trade sizes.
This work expands state-action aggregation methods for non-Markovian environments.
problem Real-world problems with large state and action spaces are not tractable with existing methods.
method Expands Extreme State Aggregation (ESA) framework to non-Markovian homomorphisms and relaxes policy uniformity.
result Near-optimal performance is guaranteed even for non-Markovian homomorphisms.
Study experiment planning with function approximation in contextual bandit problems.
problem Designing effective data collection strategies in settings with limited rewards.
method Proposes two experiment planning strategies compatible with function approximation.
result Eluder planning and sampling procedure achieves optimality guarantees.
Proposes a new sampling policy for ranking and selection problems.
problem Improving ranking and selection in adaptive sampling policies.
method Annealed entropic allocation, using soft-min weights and saddlepoint corrections.
result Consistently competitive performance in various settings.
New method for estimating value of optimal policies in uncertain scenarios.
problem Inference for optimal policies when they are non-unique or nearly deterministic.
method Semiparametric efficiency bound, uniformly weighted estimator, NSAVE method.
result Proposes NSAVE method for robust inference in uncertain optimal policies.
Algorithm optimizes electricity procurement costs by 1.65%.
problem Minimizing energy cost while covering forecast consumption.
method Deep learning forecasting and deviation indicator.
result Reduction of 1.65% in costs compared to uniform policy.
Softmax policy gradient methods converge at O ( 1 / t ) O(1/t) O ( 1/ t ) rate with constants depending on problem and initialization.
problem Understanding convergence rates of softmax policy gradient methods in tabular settings.
method Analysis of softmax policy gradient and entropy regularized policy gradient methods, using Łojasiewicz inequality and lower bounds.
result Entropy regularization improves convergence rate from O ( 1 / t ) O(1/t) O ( 1/ t ) to O ( e − c ⋅ t ) O(e^{-c \cdot t}) O ( e − c ⋅ t ) . Unified framework for risk-aware policy learning in contextual bandits.
problem Optimizing decision rules in high-stakes domains with adverse outcomes.
method Distributional framework for Lipschitz-continuous risk functionals, with novel empirical concentration inequalities.
result Data-dependent suboptimality bounds with an i l d e O ( 1 / n ) ilde{\mathcal{O}}(1/\sqrt{n}) i l d e O ( 1/ n ) rate, matching risk-neutral offline policy optimization. We study the problem of learning personalized decision policies from observational data while accounting for possible unobserved confounding. Previous approaches, which assume unconfoundedness, i.e., that no unobserved confounders affect both the treatment assignment as well as outcome, can lead to policies that introd…
New algorithm reduces best-in-class regret in contextual bandits.
problem Compete with the best policy in a class without model restrictions.
method Proposes an algorithm that updates policies by minimizing a pessimistic objective, including a clipped inverse-propensity estimate and variance penalty.
result Achieves fast best-in-class regret rates, including polylogarithmic rates in the parametric case.
Prognosticator improves performance in non-stationary MDPs.
problem Real-world applications often violate the stationary assumption in reinforcement learning.
method A policy gradient algorithm that maximizes a forecast of future performance.
result Prognosticator outperforms online adaptation techniques in non-stationary MDPs.
We study the problem of offline policy optimization in stochastic contextual bandit problems, where the goal is to learn a near-optimal policy based on a dataset of decision data collected by a suboptimal behavior policy. Rather than making any structural assumptions on the reward function, we assume access to a given …
Paper analyzes NAC with neural networks for efficient policy optimization.
problem Improving sample and iteration complexity in policy optimization.
method Entropy regularization, averaging, neural network approximation, and optimization techniques.
result Entropy regularization and averaging ensure stability and sharp sample complexity bounds.
This study explores how choosing noninformative priors affects Thompson Sampling in multiparameter bandit models.
problem The optimality of Thompson Sampling (TS) in multiparameter bandit models depends on the choice of priors, especially when models are complex.
method The study extends regret analysis to uniform distributions and proposes a modified TS policy, TS-T, to achieve asymptotic optimality.
result Changing noninformative priors can significantly affect the expected regret in multiparameter bandit models.
We study the problem of learning Markov decision processes with finite state and action spaces when the transition probability distributions and loss functions are chosen adversarially and are allowed to change with time. We introduce an algorithm whose regret with respect to any policy in a comparison class grows as t…
New method learns nonlinear systems from single finite trajectory samples.
problem Learning stabilizable nonlinear systems from single finite trajectory samples.
method Gradient-based algorithms with noise-sensitive uniform convergence guarantees.
result Efficient learning of general nonlinear systems with high accuracy and small sample complexity.
CQL (ReDS) learns from varied driving behaviors, improving offline RL performance.
problem Learning from datasets with non-uniform variability in behavior policies.
method Reweighting the data distribution to allow per-state flexibility in following the behavior policy.
result CQL (ReDS) improves performance in various offline RL tasks.
AES improves policy gradient performance by adaptively selecting experience.
problem High variance in gradient estimators from past trajectories.
method AES learns an adaptive sampling distribution to minimise gradient variance.
result AES leads to significantly improved performance in continuous control tasks.