Bayesian REX learns Atari games from demonstrations efficiently.
problem Bayesian reward learning for complex control problems is computationally intractable.
method Bayesian Reward Extrapolation (Bayesian REX) pre-trains a low-dimensional feature encoding and uses preferences to perform fast Bayesian inference.
result Bayesian REX learns Atari games from demonstrations in 5 minutes, competitive with state-of-the-art methods.
Bayesian inverse reinforcement learning (IRL) methods are ideal for safe imitation learning, as they allow a learning agent to reason about reward uncertainty and the safety of a learned policy. However, Bayesian IRL is computationally intractable for high-dimensional problems because each sample from the posterior req…
PROWL uses robust reward estimates to improve ITR selection.
problem Reward uncertainty in ITR estimation leads to inflated performance.
method PAC-Bayesian framework with reward uncertainty certificates.
result PROWL achieves better robust treatment regime estimation.
Bayesian Robust Optimization for Imitation Learning (BROIL) balances risk and reward.
problem Learning robust policies for new states in imitation learning.
method Bayesian reward function inference and user-specific risk tolerance.
result BROIL outperforms risk-sensitive and risk-neutral algorithms.
We here adopt Bayesian nonparametric mixture models to extend multi-armed bandits in general, and Thompson sampling in particular, to scenarios where there is reward model uncertainty. In the stochastic multi-armed bandit, the reward for the played arm is generated from an unknown distribution. Reward uncertainty, i.e.…
Adversarial CBO optimizes under interventions by adversaries and non-stationarities.
problem Optimizing in the presence of adversaries and non-stationary factors.
method Formalizes CBO as ACBO, introduces CBO-MW algorithm combining online learning and causal modeling.
result First algorithm with bounded regret for ACBO, achieving superior performance in synthetic and real-world environments.
The explore{exploit dilemma is one of the central challenges in Reinforcement Learning (RL). Bayesian RL solves the dilemma by providing the agent with information in the form of a prior distribution over environments; however, full Bayesian planning is intractable. Planning with the mean MDP is a common myopic approxi…
This paper presents a novel nonmyopic adaptive Gaussian process planning (GPP) framework endowed with a general class of Lipschitz continuous reward functions that can unify some active learning/sensing and Bayesian optimization criteria and offer practitioners some flexibility to specify their desired choices for defi…
We extend Bayesian multi-armed bandit (MAB) algorithms beyond their original setting by making use of sequential Monte Carlo (SMC) methods. A MAB is a sequential decision making problem where the goal is to learn a policy that maximizes long term payoff, where only the reward of the executed action is observed. In the …
We propose a generic, Bayesian, information geometric approach to the exploration--exploitation trade-off in multi-armed bandit problems. Our approach, BelMan, uniformly supports pure exploration, exploration--exploitation, and two-phase bandit problems. The knowledge on bandit arms and their reward distributions is su…
The paper proves the convergence of Q-value for Gaussian rewards.
problem Existing proofs cannot guarantee convergence of the Q-function for Gaussian rewards.
method Using the central limit theorem and relaxing the condition to E[r(s,a)2]<∞. result Proves the convergence of the Q-function under the condition of E[r(s,a)2]<∞. We consider the problem of learning from sparse and underspecified rewards, where an agent receives a complex input, such as a natural language instruction, and needs to generate a complex response, such as an action sequence, while only receiving binary success-failure feedback. Such success-failure rewards are often …
ALINE optimizes Bayesian inference and data acquisition by strategically querying informative data.
problem Strategic acquisition of informative data for Bayesian inference in challenging tasks.
method Unified framework combining amortized Bayesian inference and active data acquisition using a transformer architecture trained via reinforcement learning.
result ALINE delivers both instant and accurate inference along with efficient selection of informative points.
The paper examines how updates to probabilistic models influence behavior based on evidence.
problem Understanding how updates to probabilistic models influence behavior based on evidence.
method Study of KL-regularized soft updates as Bayesian posterior updates within a single probabilistic model.
result Posterior updates determine relative incentives but not absolute rewards, which are ambiguous up to context-specific baselines.
Exploration in environments with continuous control and sparse rewards remains a key challenge in reinforcement learning (RL). Recently, surprise has been used as an intrinsic reward that encourages systematic and efficient exploration. We introduce a new definition of surprise and its RL implementation named Variation…
This paper shows RL with KL penalties is equivalent to Bayesian inference for fine-tuning LMs.
problem Fine-tuning large language models to avoid undesirable features.
method Analyzed KL-regularized RL and showed it's equivalent to variational inference.
result KL-regularized RL avoids distribution collapse and is more insightful as Bayesian inference.
This paper introduces a set of algorithms for Monte-Carlo Bayesian reinforcement learning. Firstly, Monte-Carlo estimation of upper bounds on the Bayes-optimal value function is employed to construct an optimistic policy. Secondly, gradient-based algorithms for approximate upper and lower bounds are introduced. Finally…
New algorithm optimizes long-term user satisfaction in recommendation systems.
problem Optimizing long-term user satisfaction in recommendation systems with delayed rewards.
method Developed a predictive model of delayed rewards and a bandit algorithm that balances exploration and exploitation.
result Our approach results in substantially better performance compared to short-term or delayed optimization.
Bayesian optimization improved for biased data.
problem Adversarial bias in observations, especially hidden confounders.
method Reduction to dueling bandits, information-directed sampling (IDS).
result First efficient kernelized algorithm with regret guarantees.
PFN-TS uses Thompson sampling with PFNs to improve contextual bandit performance.
problem Improving contextual bandit performance using Thompson sampling with prior-data fitted networks.
method PFN-TS converts PFN posterior predictives into mean-reward samples using a subsampled predictive central limit theorem.
result PFN-TS achieves the best average rank across nonlinear synthetic and OpenML classification-to-bandit benchmarks.
This paper optimizes driver repositioning using MARL and reward design for better service and traffic management.
problem Unserved passenger requests due to drivers' cruising behavior during passenger seeking.
method Mean field multi-agent reinforcement learning (MARL) with a reward design scheme and Bayesian optimization (BO) to solve bilevel optimization problems.
result Optimal toll charges and service charges can improve platform and city planner objectives by significant margins, leading to better traffic conditions.
Framework for optimizing search engine rankings using observational data.
problem Optimizing ranking policies for search engines using limited observational data.
method Formulated expected reward optimization problem, estimated context value distribution, trained ranking policy via Bayesian inference.
result Demonstrated trade-offs in ranking policies trained on empirical reward estimates.
A new algorithm for bandits with hierarchical rewards.
problem Learning from correlated rewards in complex hierarchies.
method Hierarchical Thompson Sampling (HierTS) for Gaussian hierarchies.
result Hierarchical Thompson Sampling reduces regret by non-constant factors in the number of actions.
Adaptive Bayesian learning agent for non-stationary bandits.
problem Non-stationary rewards in reinforcement learning.
method Dynamic memory and statistical hypothesis testing.
result Adapts to changing rewards with minimal regret.
NK bandits improve performance on nonlinear tasks.
problem Improving performance on nonlinear sequential decision tasks.
method Proposed NK bandits using neural kernels to guide policies.
result NK bandits achieve state-of-the-art performance.
DaringFed incentivizes clients in OFL with dynamic rewards under TII.
problem Designing incentives for OFL clients under dynamic, incomplete information.
method Formulated as a dynamic signaling and pricing allocation problem in a Bayesian persuasion game.
result Optimal design of DaringFed improves accuracy and convergence speed by 16.99%.
Modeling driver trajectories using inverse reinforcement learning and random utility.
problem Modeling rational driver behavior in road networks from sparse sensor data.
method Apply random utility theory to model unknown reward function, introduce extended state, and use Markov decision process.
result Maximum entropy inverse reinforcement learning is a special case of the proposed approach.
Thompson sampling used for linear bandits with normal-gamma priors.
problem Optimizing decisions in uncertain environments with linear dependencies and unknown parameters.
method Bayesian Thompson sampling with multivariate normal-gamma priors.
result Derivation of a Bayesian regret bound for the approach.
Extends Thompson sampling for RL with fewer episodes.
problem Limited episodes in RL settings.
method Batch Bayesian optimization over episodes to learn action bias terms.
result Significantly outperforms standard Thompson sampling.
Multi-agent learning is a promising method to simulate aggregate competitive behaviour in finance. Learning expert agents' reward functions through their external demonstrations is hence particularly relevant for subsequent design of realistic agent-based simulations. Inverse Reinforcement Learning (IRL) aims at acquir…
Bayesian framework for learning optimal action-value function in MDPs.
problem Uncertainty quantification in MDPs for optimal decision-making strategies.
method Full Bayesian framework including modelling, inference, and decision-making.
result Demonstrates exploration benefits of posterior sampling in MDPs.
New algorithm optimizes for long-term user satisfaction in delayed reward settings.
problem Optimizing for long-term user satisfaction in delayed reward settings.
method Developed a predictive model of delayed rewards and a bandit algorithm that combines rewards and surrogate outcomes.
result Our algorithm significantly outperforms methods that optimize for short-term proxies or rely solely on delayed rewards.
In reinforcement learning the Q-values summarize the expected future rewards that the agent will attain. However, they cannot capture the epistemic uncertainty about those rewards. In this work we derive a new Bellman operator with associated fixed point we call the `knowledge values'. These K-values compress both the …
Improved Bayesian regret bound for linear Thompson sampling with general distributions.
problem Proving an improved Bayesian regret bound for linear Thompson sampling with general distributions.
method Generalized elliptical potential lemma for non-Gaussian noise and prior distributions.
result Minimax optimal regret bound for changing action sets with general prior and noise distributions.
PAC-Bayesian analysis improves lifelong learning in multi-armed bandits.
problem Improving lifelong learning in multi-armed bandits.
method PAC-Bayesian analysis for deriving lower bounds and proposing lifelong learning algorithms.
result Proposed algorithms outperform baseline methods in lifelong multi-armed bandit problems.
Proposes a real-time anomaly detection system using IRL.
problem Real-time anomaly detection in sequential data.
method Uses inverse reinforcement learning to infer reward function and evaluate anomalies.
result Effective in identifying anomalies in real-world data.
Study on Bayesian reinforcement learning performance bounds.
problem Achieving optimal performance in model-based Bayesian reinforcement learning.
method Defining minimum Bayesian regret, deriving upper bounds using relative entropy and Wasserstein distance, and applying these to specific MDP cases.
result Upper bounds on minimum Bayesian regret for MDPs, including specific cases like MAB and online optimization with partial feedback.
We tackle the problem of acting in an unknown finite and discrete Markov Decision Process (MDP) for which the expected shortest path from any state to any other state is bounded by a finite number D. An MDP consists of S states and A possible actions per state. Upon choosing an action at at state st, one re…
Proposes MCBO for causal Bayesian optimization with model learning and regret bounds.
problem Maximizing downstream variables in unknown structural models.
method Model-based causal Bayesian optimization (MCBO) that learns full system models and trades off exploration and exploitation.
result First non-asymptotic bounds for CBO and practical implementation showing superior performance.
Efficient exploration remains a challenging problem in reinforcement learning, especially for those tasks where rewards from environments are sparse. A commonly used approach for exploring such environments is to introduce some "intrinsic" reward. In this work, we focus on model uncertainty estimation as an intrinsic r…
Bayesian algorithms minimize cumulative regret in decentralized multi-agent bandits.
problem Minimizing cumulative regret in a decentralized multi-agent multi-armed bandit problem.
method Proposed decentralized Bayesian multi-armed bandit framework, including Thompson Sampling and Bayes-UCB algorithms.
result Regret scales logarithmically with constants matching those of an optimal centralized agent.
Develops RL algorithm for non-Markovian, non-stationary reward streams.
problem Maximizing rewards from non-Markovian, non-stationary reward streams.
method Uses causal DAG to construct Markov states, solves periodic MDP.
result Optimal state construction maximizes discounted rewards.
A new algorithm STE for model-based RL improves learning rates.
problem Sparse rewards and computational intractability of estimating information gain.
method Developed a novel algorithm based on Stein Information Directed Exploration (STE)E.
result Achieves sublinear Bayesian regret, outperforming prior approaches.
GACBO optimizes unknown causal graphs with interventions.
problem Optimizing a target variable on an unknown causal graph with interventions.
method Graph Agnostic Causal Bayesian Optimisation (GACBO) seeks to balance exploitation and exploration of causal structures and functions.
result GACBO outperforms baselines in simulated and real-world applications.
We address the problem of Bayesian reinforcement learning using efficient model-based online planning. We propose an optimism-free Bayes-adaptive algorithm to induce deeper and sparser exploration with a theoretical bound on its performance relative to the Bayes optimal policy, with a lower computational complexity. Th…
LF-IBIS learns optimal policies online without explicit likelihood.
problem Bayesian RL challenges due to intractable likelihood functions.
method Combines ABC with IBIS for online belief updates.
result Approximates posterior distributions for policies and parameters.
Thompson Sampling tackles noisy context in stochastic bandits.
problem Designing an action policy for noisy, corrupted contexts in stochastic bandits.
method Introducing a Thompson Sampling algorithm for Gaussian bandits with Gaussian context noise, adopting an information-theoretic analysis.
result Demonstrates the Bayesian regret of the proposed algorithm concerning the oracle's action policy.
DPPS uses DP priors for Bayesian non-parametric multi-arm bandits.
problem Optimizing multi-arm bandit environments with prior beliefs.
method Bayesian non-parametric algorithm based on Dirichlet Process priors.
result DPPS provides principled incorporation of prior beliefs and is optimal in Bayesian regret setup.