Paper explores state-action equivalence in RL, improving regret bounds.
problem Improving reinforcement learning performance by leveraging state-action equivalence.
method Introduces a notion of similarity between state-action pairs, defines equivalence structure, and presents algorithms for confidence sets.
result Confidence sets improve RL performance, especially in known equivalence structures.
New findings reveal discount regularization can be seen as a strong prior, leading to poor performance in unevenly sampled data.
problem Discount regularization leads to poor performance in unevenly sampled data.
method Equivalence theorem showing discount regularization as a strong prior, setting regularization parameters locally for individual state-action pairs.
result Discount regularization can be seen as a strong prior, leading to poor performance in unevenly sampled data.
New method for natural policy gradients converges linearly.
problem Improving natural policy gradient methods for better convergence.
method Fisher-Rao gradient flow applied to state-action distributions.
result Linear convergence rate with geometry-dependent factor.
Study risk-sensitive reinforcement learning with optimized certainty equivalents.
problem Risk-sensitive reinforcement learning in finite discounted MDPs.
method Analyzed a simple model-based approach and derived PAC sample complexity bounds.
result Established tight sample complexity bounds for value and policy learning.
Develops a dynamic mean field theory for reinforcement learning.
problem Finite state and action Bayesian reinforcement learning in large state spaces.
method Analogies with statistical physics, interpreting probabilities as couplings and values as spins, solving mean field equations.
result State-action values are statistically independent in the asymptotic state space limit, with exact or approximate equations for computation.
New RL method handles large state-action spaces with complex models.
problem Complex models and large state-action spaces in reinforcement learning.
method π-KRVI, an optimistic modification of least-squares value iteration using kernel ridge regression.
result First order-optimal regret guarantees under general settings, improving over state of the art.
New method recovers diverse policies from expert data using state-action pair weighting.
problem Recovering diverse policies from expert trajectories.
method Pointwise mutual information weighted behavioral cloning.
result Effective in focusing on state-action pairs most representative of the style.
A new RL paradigm reduces state-action-value function approximation inefficiency.
problem Challenges in state-action-value function approximation for RL.
method State Action Separable Reinforcement Learning (sasRL) decouples action space from value function learning.
result sasRL achieves up to 75% better performance than state-of-the-art MDP-based RL algorithms.
New RL method reduces sample complexity for large state-action spaces.
problem Handling large state-action spaces in RL with general Q-functions.
method Nonparametric Q-learning using kernel ridge regression.
result Sample complexity is order optimal with respect to ε and kernel complexity.
This work tackles model-based RL by optimizing state-action queries to learn policies with minimal data.
problem Expensive state transitions in practical RL problems limit the use of standard RL algorithms.
method Bayesian optimal experimental design to guide selection of state-action queries.
result Data-efficient RL approach that learns optimal policies with up to 1,000x less data.
Novel framework proves fast RL convergence in continuous spaces.
problem Analyzing stability in continuous state-action RL.
method Introduces a novel framework to analyze stability properties of RL.
result Highlights two key stability properties and demonstrates their satisfaction in RL.
New method estimates state-action stationary distribution for better off-policy policy evaluation.
problem Accurately estimating state-action stationary distribution for off-policy policy evaluation.
method Estimated Mixture Policy (EMP) for state and state-action stationary distribution corrections.
result Empirical validation shows improved accuracy over state-of-the-art methods.
Kernel-UCBVI algorithm balances exploration and exploitation in metric state-action spaces.
problem Exploration-exploitation dilemma in finite-horizon reinforcement learning with metric state-action spaces.
method Kernel-UCBVI, leveraging smoothness and kernel estimators of rewards and transitions.
result First regret bound for kernel-based RL using smoothing kernels, O(H3K2d/(2d+1)). A new method, Count-MORL, improves offline reinforcement learning by using state-action frequency.
problem Improving offline reinforcement learning performance.
method Integrates count-based conservatism into model-based offline reinforcement learning.
result The learned policy is near-optimal and outperforms existing methods.
Efficient algorithm for reinforcement learning in large state-action spaces with adaptive discretization.
problem Efficient reinforcement learning in large, potentially continuous state-action spaces.
method Adaptive Q-learning policy with data-driven adaptive discretization. result Demonstrates improved performance compared to existing methods, especially in adapting to the problem's structure.
Study optimal offline RL with uncertainty sets and distribution shifts.
problem Optimal offline reinforcement learning with limited data.
method Construct uncertainty sets and distribution shifts, solve robust Markov decision process.
result Least conservative estimator for unknown true distribution.
Hierarchical Modular Reinforcement Learning (HMRL), consists of 2 layered learning where Profit Sharing works to plan a prey position in the higher layer and Q-learning method trains the state-actions to the target in the lower layer. In this paper, we expanded HMRL to multi-target problem to take the distance between …
Develops a new method for optimizing policies in hierarchical models.
problem Optimizing complex policies in hierarchical models.
method Applies second-order methods in the space of state-action paths.
result The natural path gradient method can be computed exactly and reflects state-space hierarchy.
The paper introduces a new intrinsic reward method for exploration in reinforcement learning.
problem Improving exploration in reinforcement learning agents.
method Intrinsic rewards proportional to the entropy of future state-action features.
result The new objective leads to improved visitation of features within individual trajectories.
This paper unifies three regularization methods in batch reinforcement learning.
problem Learning overly-complex models in batch reinforcement learning.
method Unified weighted average transition matrix framework for three regularization methods.
result Empirical evaluation confirms intuitions about regularization methods' performance.
MBML improves multi-task RL by inferring task identity from state-action pairs.
problem Multi-task batch reinforcement learning with unseen tasks.
method MBML uses triplet loss and relabeling to robustify task inference.
result Significantly faster convergence on unseen tasks compared to random initialization.
BRPO optimizes batch RL policies to better exploit state-action differences.
problem Batch RL's conservatism limits exploitation of state-action differences.
method Proposes residual policies and derives BRPO to maximize policy performance.
result BRPO achieves state-of-the-art performance in various tasks.
Paper introduces SALE for better state-action learning in RL.
problem Challenges in representation learning for low-level states in RL.
method Introduces SALE, a novel approach for learning embeddings of state-action interactions.
result TD7 algorithm significantly outperforms existing continuous control algorithms.
Paper tackles covariate shift in offline IL using less proficient behavior data.
problem Mitigating covariate shift in Imitation Learning with limited data coverage.
method Model-based IL from Offline Data (MILO) framework.
result MILO can combat covariate shift even with sub-optimal behavior policy data.
SPEDER extracts state-action abstraction from dynamics for reinforcement learning.
problem Curse of dimensionality and limited applicability of spectral methods.
method Spectral Decomposition Representation (SPEDER) that extracts state-action abstraction from dynamics without policy dependence.
result Theoretical analysis establishes sample efficiency in online and offline settings.
GIM learns by analogy to improve reinforcement learning efficiency.
problem Efficient reinforcement learning with limited data.
method GIM infers unknown dynamics from known dynamics based on spectral properties.
result GIM achieves lower sample complexity and computational efficiency.
MO2 learns useful behaviours from past experience for new tasks.
problem Discovering useful behaviours from past experience and transferring them to new tasks.
method Model-Based Offline Options (MO2) framework supporting sample-efficient bottleneck option discovery over continuous state-action spaces.
result MO2 outperforms recent option learning methods on complex long-horizon continuous control tasks.
New RL approach uses future state and action visitation measures for better exploration.
problem Improving exploration in reinforcement learning.
method Intrinsic reward based on future state and action visitation measures, using contraction operators.
result Policies achieve good state-action space coverage and high performance.
We consider the problem of learning a policy for a Markov decision process consistent with data captured on the state-actions pairs followed by the policy. We assume that the policy belongs to a class of parameterized policies which are defined using features associated with the state-action pairs. The features are kno…
The Bellman error is a poor proxy for value function accuracy, even with all state-action pairs.
problem The Bellman error is a poor proxy for the accuracy of the value function.
method Study of the Bellman equation as a surrogate objective for value prediction accuracy.
result The magnitude of the Bellman error is only weakly related to the distance to the true value function, even with all state-action pairs.
Federated Q-learning achieves linear speedup with heterogeneity, improving sample complexity.
problem Collaborative learning in distributed RL settings with limited data sharing.
method Analyzes synchronous and asynchronous federated Q-learning, proposes importance averaging.
result Achieves linear speedup with heterogeneity, robust to local trajectory heterogeneity.
ZoomRL learns efficient strategies for large state-action spaces using a metric.
problem Handling large state-action spaces in reinforcement learning.
method ZoomRL leverages continuous bandits to adaptively discretize the joint space.
result Achieves worst-case regret of $ ilde{O}(H^{ rac{5}{2}} K^{ rac{d+1}{d+2}})$.
A critical and challenging problem in reinforcement learning is how to learn the state-action value function from the experience replay buffer and simultaneously keep sample efficiency and faster convergence to a high quality solution. In prior works, transitions are uniformly sampled at random from the replay buffer o…
Sample inefficiency is a long-lasting problem in reinforcement learning (RL). The state-of-the-art estimates the optimal action values while it usually involves an extensive search over the state-action space and unstable optimization. Towards the sample-efficient RL, we propose ranking policy gradient (RPG), a policy …
The paper proposes a principle for dynamically adjusting the granularity of reinforcement learning abstractions.
problem Lack of general principles for dynamically adjusting the granularity of reinforcement learning abstractions.
method The paper proposes a principle based on rate-distortion theory, formalized through a performance certificate decomposing value error into learning and abstraction error bounds.
result Soft state-action abstractions can achieve near-optimal performance under substantial lossy compression of state and action information.
The paper improves Q-learning by incorporating pessimism for better sample efficiency.
problem Improving sample efficiency in asynchronous Q-learning with non-i.i.d. data.
method Developed an algorithmic framework that incorporates the principle of pessimism into asynchronous Q-learning, penalizing infrequently-visited state-action pairs based on suitable lower confidence bounds (LCBs).
result Achieved near-optimal sample complexity, providing theoretical support for the use of pessimism in non-i.i.d. data.
QR-MIX models joint state-action values as a distribution to handle randomness in MARL.
problem Randomness in rewards and observations leads to randomness in long-term returns in MARL.
method QR-MIX uses quantile regression and combines it with QMIX and IQN to model joint state-action values as a distribution.
result QR-MIX outperforms QMIX in the StarCraft Multi-Agent Challenge (SMAC) environment.
Estimation of importance sampling weights for off-policy evaluation of contextual bandits often results in imbalance - a mismatch between the desired and the actual distribution of state-action pairs after weighting. In this work we present balanced off-policy evaluation (B-OPE), a generic method for estimating weights…
KeRNS tackles non-stationary reinforcement learning in metric spaces.
problem Non-stationary reinforcement learning in metric spaces.
method KeRNS uses time-dependent kernels to model non-stationary Markov Decision Processes (MDPs).
result KeRNS achieves a regret bound that scales with the covering dimension and total variation of the MDP.
RPO uses past and future state-action info for better policy optimization.
problem Sample inefficiency in on-policy reinforcement learning methods.
method Reflective Policy Optimization (RPO) integrates past and future state-action info for policy improvement.
result RPO improves policy performance and contracts the solution space, leading to faster convergence.
Model-free learning for multi-agent stochastic games is an active area of research. Existing reinforcement learning algorithms, however, are often restricted to zero-sum games, and are applicable only in small state-action spaces or other simplified settings. Here, we develop a new data efficient Deep-Q-learning method…
Model-free Reinforcement Learning (RL) algorithms such as Q-learning [Watkins, Dayan 92] have been widely used in practice and can achieve human level performance in applications such as video games [Mnih et al. 15]. Recently, equipped with the idea of optimism in the face of uncertainty, Q-learning algorithms [Jin, Al…
New framework for RL transfer learning with state-action mismatch.
problem High sample complexity in RL from scratch.
method Embeddings to transfer knowledge between MDPs with different state- and action-spaces.
result Successful transfer learning in scenarios with state- and action-space mismatches.
Develops a new reinforcement learning framework for complex control problems.
problem Continuous-time extended mean field control with deterministic policies.
method Model-free sensitivity formula, deterministic policy gradient, local value and advantage-rate representations.
result Demonstrates efficiency, stability, and robustness in solving complex control problems.
Paper learns meaningful state and action representations from MDP trajectories.
problem Learning good state and action representations from MDP trajectories.
method Tensor decomposition, kernelization, importance sampling, low-Tucker-rank approximation.
result The learned state/action abstractions provide accurate approximations to latent block structures.
OpenAI Five defeated Dota 2 champions using deep reinforcement learning.
problem Complex, continuous state-action spaces in Dota 2.
method Deep reinforcement learning, distributed training, self-play.
result Superhuman performance in Dota 2 achieved through reinforcement learning.
Most real-world problems have huge state and/or action spaces. Therefore, a naive application of existing tabular solution methods is not tractable on such problems. Nonetheless, these solution methods are quite useful if an agent has access to a relatively small state-action space homomorphism of the true environment …
This paper introduces CENIE to quantify environment novelty for better UED.
problem Challenges in measuring environment novelty for effective UED.
method CENIE framework using state-action space coverage and Gaussian Mixture Models.
result CENIE improves UED performance across multiple benchmarks.