New method estimates state-action stationary distribution for better off-policy policy evaluation.
problem Accurately estimating state-action stationary distribution for off-policy policy evaluation.
method Estimated Mixture Policy (EMP) for state and state-action stationary distribution corrections.
result Empirical validation shows improved accuracy over state-of-the-art methods.
MetaCURL tackles non-stationary MDPs with optimal dynamic regret.
problem Online learning in non-stationary Markov decision processes.
method MetaCURL uses a meta-algorithm with multiple black-box algorithms and a sleeping expert framework.
result Achieves optimal dynamic regret without prior knowledge of MDP changes.
In many real-world reinforcement learning applications, access to the environment is limited to a fixed dataset, instead of direct (online) interaction with the environment. When using this data for either evaluation or training of a new policy, accurate estimates of discounted stationary distribution ratios -- correct…
KeRNS tackles non-stationary reinforcement learning in metric spaces.
problem Non-stationary reinforcement learning in metric spaces.
method KeRNS uses time-dependent kernels to model non-stationary Markov Decision Processes (MDPs).
result KeRNS achieves a regret bound that scales with the covering dimension and total variation of the MDP.
Non-stationary reinforcement learning is challenging due to the complexity of updating value functions.
problem Challenges in non-stationary reinforcement learning, especially in updating value functions.
method Proved a worst-case complexity result for modifying reinforcement learning problems.
result Modifying reinforcement learning problems requires an amount of time almost as large as the number of states.
New method stabilizes FQE by reweighting Bellman targets.
problem Stability guarantees for FQE often rely on Bellman completeness, which can fail with function approximation.
method Proposes stationary-weighted FQE, reweighting Bellman targets by stationary target-to-behavior density ratio.
result Proves finite-sample linear convergence to stationary projected Bellman fixed point without Bellman completeness.
Reward suffices for convex MDPs, expanding RL to new problems.
problem Capturing goals as convex functions of stationary distribution.
method Reformulated as a min-max game using Fenchel duality.
result Convex MDPs require non-stationary reward functions.
New method improves stability of soft FQI for offline RL.
problem Stability issues in soft FQI under function approximation.
method Stationary reweighting to align operator norms.
result Local linear convergence proved under certain conditions.
We consider the problem of learning a policy for a Markov decision process consistent with data captured on the state-actions pairs followed by the policy. We assume that the policy belongs to a class of parameterized policies which are defined using features associated with the state-action pairs. The features are kno…
GWIL uses Gromov-Wasserstein distance to align expert and imitation agent states.
problem Cross-domain imitation learning challenges due to different system dimensions and stationary distributions.
method Gromov-Wasserstein Imitation Learning (GWIL) using Gromov-Wasserstein distance.
result GWIL effectively aligns expert and imitation agent states in various continuous control domains.
Federated Q-learning achieves linear speedup with heterogeneity, improving sample complexity.
problem Collaborative learning in distributed RL settings with limited data sharing.
method Analyzes synchronous and asynchronous federated Q-learning, proposes importance averaging.
result Achieves linear speedup with heterogeneity, robust to local trajectory heterogeneity.
Develops a new method for optimizing policies in hierarchical models.
problem Optimizing complex policies in hierarchical models.
method Applies second-order methods in the space of state-action paths.
result The natural path gradient method can be computed exactly and reflects state-space hierarchy.
New method for natural policy gradients converges linearly.
problem Improving natural policy gradient methods for better convergence.
method Fisher-Rao gradient flow applied to state-action distributions.
result Linear convergence rate with geometry-dependent factor.
Study optimal offline RL with uncertainty sets and distribution shifts.
problem Optimal offline reinforcement learning with limited data.
method Construct uncertainty sets and distribution shifts, solve robust Markov decision process.
result Least conservative estimator for unknown true distribution.
The paper introduces a new intrinsic reward method for exploration in reinforcement learning.
problem Improving exploration in reinforcement learning agents.
method Intrinsic rewards proportional to the entropy of future state-action features.
result The new objective leads to improved visitation of features within individual trajectories.
MBML improves multi-task RL by inferring task identity from state-action pairs.
problem Multi-task batch reinforcement learning with unseen tasks.
method MBML uses triplet loss and relabeling to robustify task inference.
result Significantly faster convergence on unseen tasks compared to random initialization.
A critical and challenging problem in reinforcement learning is how to learn the state-action value function from the experience replay buffer and simultaneously keep sample efficiency and faster convergence to a high quality solution. In prior works, transitions are uniformly sampled at random from the replay buffer o…
New RL approach uses future state and action visitation measures for better exploration.
problem Improving exploration in reinforcement learning.
method Intrinsic reward based on future state and action visitation measures, using contraction operators.
result Policies achieve good state-action space coverage and high performance.
New algorithm reduces offline RL data requirements significantly.
problem Optimizing policies using only historical data in reinforcement learning.
method Off-Policy Double Variance Reduction (OPDVR) algorithm.
result OPDVR achieves optimal sample complexity with O(H2/dmε2) episodes. QR-MIX models joint state-action values as a distribution to handle randomness in MARL.
problem Randomness in rewards and observations leads to randomness in long-term returns in MARL.
method QR-MIX uses quantile regression and combines it with QMIX and IQN to model joint state-action values as a distribution.
result QR-MIX outperforms QMIX in the StarCraft Multi-Agent Challenge (SMAC) environment.
When using reinforcement learning (RL) algorithms to evaluate a policy it is common, given a large state space, to introduce some form of approximation architecture for the value function (VF). The exact form of this architecture can have a significant effect on the accuracy of the VF estimate, however, and determining…
New findings reveal discount regularization can be seen as a strong prior, leading to poor performance in unevenly sampled data.
problem Discount regularization leads to poor performance in unevenly sampled data.
method Equivalence theorem showing discount regularization as a strong prior, setting regularization parameters locally for individual state-action pairs.
result Discount regularization can be seen as a strong prior, leading to poor performance in unevenly sampled data.
This paper unifies three regularization methods in batch reinforcement learning.
problem Learning overly-complex models in batch reinforcement learning.
method Unified weighted average transition matrix framework for three regularization methods.
result Empirical evaluation confirms intuitions about regularization methods' performance.
Develops a new reinforcement learning framework for complex control problems.
problem Continuous-time extended mean field control with deterministic policies.
method Model-free sensitivity formula, deterministic policy gradient, local value and advantage-rate representations.
result Demonstrates efficiency, stability, and robustness in solving complex control problems.
BRPO optimizes batch RL policies to better exploit state-action differences.
problem Batch RL's conservatism limits exploitation of state-action differences.
method Proposes residual policies and derives BRPO to maximize policy performance.
result BRPO achieves state-of-the-art performance in various tasks.
Estimation of importance sampling weights for off-policy evaluation of contextual bandits often results in imbalance - a mismatch between the desired and the actual distribution of state-action pairs after weighting. In this work we present balanced off-policy evaluation (B-OPE), a generic method for estimating weights…
IDAC improves reinforcement learning efficiency by modeling implicit distributions.
problem Improving sample efficiency in reinforcement learning algorithms.
method IDAC uses two DGNs for a distributional critic and a semi-implicit actor to model implicit policy distributions.
result IDAC outperforms state-of-the-art algorithms on OpenAI Gym environments.
Paper explores state-action equivalence in RL, improving regret bounds.
problem Improving reinforcement learning performance by leveraging state-action equivalence.
method Introduces a notion of similarity between state-action pairs, defines equivalence structure, and presents algorithms for confidence sets.
result Confidence sets improve RL performance, especially in known equivalence structures.
The method approximates stationary distributions of Markov models by truncating irrelevant states.
problem Computing the stationary distribution of complex Markov models is computationally challenging.
method A state-space lumping scheme that aggregates states in a grid structure, iteratively refining the state-space.
result The method provides a well-justified finite-state projection tailored to the stationary behavior of Markov models.
Proposes a method to forecast non-stationary time series.
problem Challenges of non-stationary conditional distributions in deep learning.
method Bayesian dynamic model + deep conditional distribution model.
result Adapts to non-stationary time series better than state-of-the-art solutions.
New RL method handles large state-action spaces with complex models.
problem Complex models and large state-action spaces in reinforcement learning.
method π-KRVI, an optimistic modification of least-squares value iteration using kernel ridge regression.
result First order-optimal regret guarantees under general settings, improving over state of the art.
A new method estimates multi-dimensional value distributions using Hilbert space embeddings.
problem Estimating value distributions in complex, multi-dimensional reinforcement learning settings.
method Hilbert space mappings and kernel mean embeddings to estimate the kernel mean embedding of multi-dimensional value distributions.
result Uniform convergence guarantees and robust off-policy evaluation demonstrated in simulations.
The paper provides exact multivariate amplitude distributions for non-stationary Gaussian or algebraic fluctuations.
problem Capturing the statistical properties of fluctuating correlations in non-stationary systems.
method Developed a random matrix model to average multivariate amplitude distributions from short time scales to large time scales.
result Explicit multivariate distributions for non-stationary correlation systems are provided, capturing the degree of non-stationarity.
New method recovers diverse policies from expert data using state-action pair weighting.
problem Recovering diverse policies from expert trajectories.
method Pointwise mutual information weighted behavioral cloning.
result Effective in focusing on state-action pairs most representative of the style.
A new RL paradigm reduces state-action-value function approximation inefficiency.
problem Challenges in state-action-value function approximation for RL.
method State Action Separable Reinforcement Learning (sasRL) decouples action space from value function learning.
result sasRL achieves up to 75% better performance than state-of-the-art MDP-based RL algorithms.
New RL method reduces sample complexity for large state-action spaces.
problem Handling large state-action spaces in RL with general Q-functions.
method Nonparametric Q-learning using kernel ridge regression.
result Sample complexity is order optimal with respect to ε and kernel complexity.
This work tackles model-based RL by optimizing state-action queries to learn policies with minimal data.
problem Expensive state transitions in practical RL problems limit the use of standard RL algorithms.
method Bayesian optimal experimental design to guide selection of state-action queries.
result Data-efficient RL approach that learns optimal policies with up to 1,000x less data.
Improved sampling from mean-field stationary distributions.
problem Sampling from the stationary distribution of mean-field SDEs.
method Decoupling the problem into two aspects: approximation of mean-field SDE and sampling from finite-particle distribution.
result Improved guarantees in various settings, including optimizing neural networks.
Novel framework proves fast RL convergence in continuous spaces.
problem Analyzing stability in continuous state-action RL.
method Introduces a novel framework to analyze stability properties of RL.
result Highlights two key stability properties and demonstrates their satisfaction in RL.
New method reduces computational cost for learning stationary diffusions.
problem Learning parameters of stationary diffusions efficiently.
method Stein-type discrepancy (SKDS) for estimating generator expectations.
result SKDS guarantees alignment with target stationary distribution.
GenDICE estimates stationary values from offline data.
problem Estimating stationary values from limited offline data.
method Ratio correction based on stationary distribution properties, variational divergence minimization.
result Consistent estimation of stationary values possible in offline settings.
In this work, we build on recent advances in distributional reinforcement learning to give a generally applicable, flexible, and state-of-the-art distributional variant of DQN. We achieve this by using quantile regression to approximate the full quantile function for the state-action return distribution. By reparameter…
New SDE model from machine learning optimization with unique stationary distribution.
problem Stationary distribution of machine learning optimization models.
method Proved ergodicity and unique stationary distribution of power-law dynamic SDE.
result Power-law dynamic has a unique stationary distribution and is ergodic.
Study best arm identification in restless bandits with unknown TPMs.
problem Identify the best arm with fixed confidence in restless bandits with unknown TPMs.
method Proposed a policy for best arm identification and proved its expected stopping time matches the lower bound.
result The state-action visitation proportions match the optimal proportions under any asymptotically optimal policy.
Develops CLTs for Markov chain transition probabilities and policies.
problem Estimating transition probabilities and policies in controlled Markov chains.
method Non-parametric estimator for transition matrices; CLTs for value, Q-, and advantage functions; goodness-of-fit tests.
result Asymptotic normality of estimators under specific logging policies.
Estimates stationary distribution from batch transitions without access to the underlying process.
problem Estimating stationary distribution from batch transitions without access to the underlying process.
method Proposes a consistent estimator based on a correction ratio function and variational power method (VPM).
result VPM provides significantly better estimates across various problems.
New algorithm for online learning in episodic MDPs with convex objectives.
problem Online episodic convex reinforcement learning.
method Online mirror descent algorithm with varying constraint sets and exploration bonus.
result Near-optimal regret bounds for online CURL without prior knowledge of transition function.
Develops a dynamic mean field theory for reinforcement learning.
problem Finite state and action Bayesian reinforcement learning in large state spaces.
method Analogies with statistical physics, interpreting probabilities as couplings and values as spins, solving mean field equations.
result State-action values are statistically independent in the asymptotic state space limit, with exact or approximate equations for computation.