arXiv research

Novel IRL method identifies suboptimal medical decisions in ICU data.

Inferring intent from observed behavior has been studied extensively within the frameworks of Bayesian inverse planning and inverse reinforcement learning. These methods infer a goal or reward function that best explains the actions of the observed agent, typically a human demonstrator. Another agent can use this infer…

New study shows exponential lower bound for RL even with constant suboptimality gap.

MaxMax Q-Learning improves coordination in multi-agent reinforcement learning by refining action selection.

New algorithm reduces regret by focusing on near-optimal actions.

Curriculum learning has been successfully used in reinforcement learning to accelerate the learning process, through knowledge transfer between tasks of increasing complexity. Critical tasks, in which suboptimal exploratory actions must be minimized, can benefit from curriculum learning, and its ability to shape explor…

Improved reinforcement learning for episodes with varying action sets.

The Bellman error is a poor proxy for value function accuracy, even with all state-action pairs.

Paper explores limits of imitation learning in MDPs, setting new suboptimality bounds.

New method identifies flawed internal models of the world in animals.

The exploration mechanism used by a Deep Reinforcement Learning (RL) agent plays a key role in determining its sample efficiency. Thus, improving over random exploration is crucial to solve long-horizon tasks with sparse rewards. We propose to leverage an ensemble of partial solutions as teachers that guide the agent's…

A reinforcement learning agent tries to maximize its cumulative payoff by interacting in an unknown environment. It is important for the agent to explore suboptimal actions as well as to pick actions with highest known rewards. Yet, in sensitive domains, collecting more data with exploration is not always possible, but…

New algorithms tackle robust RL with linear models, revealing unique challenges.

Safe linear bandits over unknown polytopes avoid safety violations and suboptimal actions.

Most methods for decision-theoretic online learning are based on the Hedge algorithm, which takes a parameter called the learning rate. In most previous analyses the learning rate was carefully tuned to obtain optimal worst-case performance, leading to suboptimal performance on easy instances, for example when there ex…

Two entropy measures quantify suboptimal portfolio performance.

We consider the restless Markov bandit problem, in which the state of each arm evolves according to a Markov process independently of the learner's actions. We suggest an algorithm that after $T$ steps achieves $\tilde{O}(\sqrt{T})$ regret with respect to the best policy that knows the distributions of all arms. No ass…

Reinforcement learning with sparse rewards is still an open challenge. Classic methods rely on getting feedback via extrinsic rewards to train the agent, and in situations where this occurs very rarely the agent learns slowly or cannot learn at all. Similarly, if the agent receives also rewards that create suboptimal m…

Policy gradient methods have enjoyed great success in deep reinforcement learning but suffer from high variance of gradient estimates. The high variance problem is particularly exasperated in problems with long horizons or high-dimensional action spaces. To mitigate this issue, we derive a bias-free action-dependent ba…

Boltzmann exploration is a classic strategy for sequential decision-making under uncertainty, and is one of the most standard tools in Reinforcement Learning (RL). Despite its widespread use, there is virtually no theoretical understanding about the limitations or the actual benefits of this exploration scheme. Does it…

We address reinforcement learning problems with finite state and action spaces where the underlying MDP has some known structure that could be potentially exploited to minimize the exploration rates of suboptimal (state, action) pairs. For any arbitrary structure, we derive problem-specific regret lower bounds satisfie…

In the context of tree-search stochastic planning algorithms where a generative model is available, we consider on-line planning algorithms building trees in order to recommend an action. We investigate the question of avoiding re-planning in subsequent decision steps by directly using sub-trees as action recommender. …

New algorithm reduces MDP regret by accounting for state suboptimality gaps and variance.

SAVO actor improves reinforcement learning by avoiding local optima in complex Q-functions.

FinFlowRL learns from experts to optimize financial control in changing markets.

Method trains neural network for optimal decisions from stochastic simulators.

POLAR optimizes treatment strategies in dynamic settings with statistical guarantees.

New algorithm learns policies without uniform overlap assumption.

Novel ML approach solves complex warehouse routing problem.

New suboptimal algorithm for best subset selection in high-dimensional data.

Paper tackles RLHF with DCPPO method, proving near-optimal suboptimality.

Policy gradient methods have demonstrated success in reinforcement learning tasks that have high-dimensional continuous state and action spaces. However, policy gradient methods are also notoriously sample inefficient. This can be attributed, at least in part, to the high variance in estimating the gradient of the task…

The construction by Du et al. (2019) implies that even if a learner is given linear features in $\mathbb R^d$ that approximate the rewards in a bandit with a uniform error of $ε$, then searching for an action that is optimal up to $O(ε)$ requires examining essentially all actions. We use the Kiefer-Wolfowitz theorem to…

The maximum a posteriori (MAP) configuration of binary variable models with submodular graph-structured energy functions can be found efficiently and exactly by graph cuts. Max-product belief propagation (MP) has been shown to be suboptimal on this class of energy functions by a canonical counterexample where MP conver…

New calibration measure SCDL improves trust in AI predictions.

A new algorithm CAP learns optimal policies from observational data with confounding bias and missing observations.

PFPN uses particle filtering to improve character control in physics-based simulations.

New algorithm tackles confounding in offline RL for partially observable MDPs.

Markov Decision Processes (MDPs), the mathematical framework underlying most algorithms in Reinforcement Learning (RL), are often used in a way that wrongfully assumes that the state of an agent's environment does not change during action selection. As RL systems based on MDPs begin to find application in real-world sa…

Diffusion-QL uses diffusion models to improve offline RL performance.

Curriculum learning in reinforcement learning is used to shape exploration by presenting the agent with increasingly complex tasks. The idea of curriculum learning has been largely applied in both animal training and pedagogy. In reinforcement learning, all previous task sequencing methods have shaped exploration with …

Study shows how to learn optimal policies quickly in stochastic control problems.

We address online linear optimization problems when the possible actions of the decision maker are represented by binary vectors. The regret of the decision maker is the difference between her realized loss and the best loss she would have achieved by picking, in hindsight, the best possible action. Our goal is to unde…

A new algorithm for offline RL with trajectory-wise reward reduces bias and variance errors.

The Lasso is suboptimal in sparse linear regression due to design matrix constraints.

The paper improves Q-learning by incorporating pessimism for better sample efficiency.

Off-policy reinforcement learning aims to leverage experience collected from prior policies for sample-efficient learning. However, in practice, commonly used off-policy approximate dynamic programming methods based on Q-learning and actor-critic methods are highly sensitive to the data distribution, and can make only …

New bounds on adaptivity cost in stochastic optimization.