We study an exploration method for model-free RL that generalizes the counter-based exploration bonus methods and takes into account long term exploratory value of actions rather than a single step look-ahead. We propose a model-free RL method that modifies Delayed Q-learning and utilizes the long-term exploration bonu…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Bonus Exploration”
Study minimax optimal RL in factored MDPs with bonus exploration.
problem Optimal reinforcement learning in episodic factored MDPs.
method Proposes two model-based algorithms with bonus exploration for minimax optimal regret.
result Achieves minimax optimal regret guarantees for rich factored structures.
We introduce an exploration bonus for deep reinforcement learning methods that is easy to implement and adds minimal overhead to the computation performed. The bonus is the error of a neural network predicting features of the observations given by a fixed randomly initialized neural network. We also introduce a method …
We introduce and analyse two algorithms for exploration-exploitation in discrete and continuous Markov Decision Processes (MDPs) based on exploration bonuses. SCAL is a variant of SCAL (Fruit et al., 2018) that performs efficient exploration-exploitation in any unknown weakly-communicating MDP for which an upper bo…
This paper provides an empirical evaluation of recently developed exploration algorithms within the Arcade Learning Environment (ALE). We study the use of different reward bonuses that incentives exploration in reinforcement learning. We do so by fixing the learning algorithm used and focusing only on the impact of the…
EQO uses a simple bonus term for efficient exploration in tabular RL.
problem Achieving minimax optimal reinforcement learning with practical efficiency.
method Exploration via Quasi-Optimism, employing a bonus proportional to state-action visit count.
result Achieves the sharpest known regret bound for tabular RL.
Curriculum learning speeds up agent learning in Minecraft, a complex visual domain.
problem Training agents to learn multiple tasks in a complex, visual domain.
method Learning-progress based curriculum and dynamic exploration bonuses.
result Curriculum learning improves agent performance in a complex reinforcement learning problem.
Although exploration in reinforcement learning is well understood from a theoretical point of view, provably correct methods remain impractical. In this paper we study the interplay between exploration and approximation, what we call approximate exploration. Our main goal is to further our theoretical understanding of …
Bonus-malus Systems vs Delays in Claims Reporting and Settlement: Analysis of Ruin Probabilitiesq-fin.RM
The study analyzes how bonus-malus systems and delayed claims settlement affect insurance companies' financial stability.
problem Analyzing the impact of bonus-malus systems and delayed claims settlement on insurance companies' financial stability.
method Examined a discrete-time risk model with time-varying premiums, evaluating two types of claims and settlement delays.
result Delayed settlement of by-claims leads to lower ruin probabilities under specific assumptions.
Study shows how to motivate AI agents like humans through learning from demonstrations.
problem Transfer human-like intrinsic motivation to AI agents in RL.
method Inverse RL approach to learn an exploration bonus from demonstrations.
result AI agents can learn and use complex exploration behaviors from demonstrations.
The paper deals with bonus-malus systems with different claim types and varying deductibles. The premium relativities are softened for the policyholders who are in the malus zone and these policyholders are subject to per claim deductibles depending on their levels in the bonus-malus scale and the types of the reported…
The paper introduces Bellman-consistent pessimism to improve offline reinforcement learning without overly pessimistic bias.
problem Offline reinforcement learning's challenge of discovering good policies without exhaustive exploration.
method Introduces Bellman-consistent pessimism for function approximation, improving sample complexity and adaptability.
result Improves sample complexity by in the action space finite case, and automatically adapts to bias-variance tradeoff.
The paper calculates bonus values in complex insurance schemes.
problem Calculating bonus payments in multi-state with-profit life insurance.
method Combines financial risk simulation with insurance risk methods.
result Efficient numerical procedures for bonus calculation.
Proposes EVE for efficient exploration in reinforcement learning.
problem Efficient exploration in reinforcement learning.
method EVE: a recipe for posterior over parameters, facilitating efficient exploration.
result Competitive performance on benchmarks, efficient exploration confirmed.
Develops a Bonus-Malus model for cyber risk insurance to incentivize cybersecurity.
problem Lack of effective insurance strategies to incentivize cybersecurity.
method Proposes a Bonus-Malus model and a mathematical model with a numerical algorithm.
result Demonstrates how a Bonus-Malus system resolves moral hazard and benefits the insurer.
Unified framework for distributional regret in bandits and reinforcement learning.
problem Characterizing the distribution of regret in multi-armed bandits and reinforcement learning.
method Unified framework with a UCBVI-style algorithm and distributional regret bounds.
result Distributional regret bounds with optimal trade-offs between expected and distributional regret.
In distributional reinforcement learning (RL), the estimated distribution of value function models both the parametric and intrinsic uncertainties. We propose a novel and efficient exploration method for deep RL that has two components. The first is a decaying schedule to suppress the intrinsic uncertainty. The second …
Efficient exploration is necessary to achieve good sample efficiency for reinforcement learning in general. From small, tabular settings such as gridworlds to large, continuous and sparse reward settings such as robotic object manipulation tasks, exploration through adding an uncertainty bonus to the reward function ha…
In recent years, deep reinforcement learning has been shown to be adept at solving sequential decision processes with high-dimensional state spaces such as in the Atari games. Many reinforcement learning problems, however, involve high-dimensional discrete action spaces as well as high-dimensional state spaces. This pa…
A fundamental question in reinforcement learning is whether model-free algorithms are sample efficient. Recently, Jin et al. \cite{jin2018q} proposed a Q-learning algorithm with UCB exploration policy, and proved it has nearly optimal regret bound for finite-horizon episodic MDP. In this paper, we adapt Q-learning with…
The explore{exploit dilemma is one of the central challenges in Reinforcement Learning (RL). Bayesian RL solves the dilemma by providing the agent with information in the form of a prior distribution over environments; however, full Bayesian planning is intractable. Planning with the mean MDP is a common myopic approxi…
Exploration strategy design is one of the challenging problems in reinforcement learning~(RL), especially when the environment contains a large state space or sparse rewards. During exploration, the agent tries to discover novel areas or high reward~(quality) areas. In most existing methods, the novelty and quality in …
An agent learning through interactions should balance its action selection process between probing the environment to discover new rewards and using the information acquired in the past to adopt useful behaviour. This trade-off is usually obtained by perturbing either the agent's actions (e.g., e-greedy or Gibbs sampli…
Exploration in sparse reward reinforcement learning remains an open challenge. Many state-of-the-art methods use intrinsic motivation to complement the sparse extrinsic reward signal, giving the agent more opportunities to receive feedback during exploration. Commonly these signals are added as bonus rewards, which res…
We discuss the pricing methodology for Bonus Certificates and Barrier Reverse-Convertible Structured Products. Pricing for a European barrier condition is straightforward for products of both types and depends on an efficient interpolation of observed market option pricing. Pricing products We discuss the pricing metho…
In this work we present a novel approach for transfer-guided exploration in reinforcement learning that is inspired by the human tendency to leverage experiences from similar encounters in the past while navigating a new task. Given an optimal policy in a related task-environment, we show that its bisimulation distance…
Rewards are sparse in the real world and most of today's reinforcement learning algorithms struggle with such sparsity. One solution to this problem is to allow the agent to create rewards for itself - thus making rewards dense and more suitable for learning. In particular, inspired by curious behaviour in animals, obs…
New exploration bonuses improve reinforcement learning efficiency.
problem Efficient exploration in unknown environments with limited feedback.
method Improved exploration bonuses scaling with 1/n and improved stopping time analysis.
result Faster learning rates and improved sample complexity in pure-exploration settings.
New algorithm explores reinforcement learning with noisy data.
problem Exploration in reinforcement learning with complex value functions.
method Randomized exploration with i.i.d. scalar noises and optimistic reward sampling.
result Achieves worst-case regret bound of .
A Refined Analysis of UCBVIstat.ML
Improved analysis of UCBVI algorithm with better empirical performance.
problem Improving the UCBVI algorithm's performance and understanding its bounds.
method Refined analysis of UCBVI algorithm with improved bonus terms and regret analysis.
result Improving multiplicative constants in UCBVI bounds enhances empirical performance.
On Reward-Free RL with Kernel and Neural Function Approximations: Single-Agent MDP and Markov Gamecs.LG
New RL method explores environments without rewards, achieving efficient policy generation.
problem Efficiently exploring unknown environments without predefined rewards.
method Optimistic value-iteration algorithm with kernel and neural function approximations.
result Achieves sample complexity for generating policies or equilibria.
Algorithm learns robust equilibrium in online Markov games with interactive data.
problem Sim-to-real gap in reinforcement learning.
method Distributionally robust RL with minimum value assumption, least square value iteration.
result Sample-efficient algorithm for robust equilibrium in online Markov games.
In this paper we introduce a simple approach for exploration in reinforcement learning (RL) that allows us to develop theoretically justified algorithms in the tabular case but that is also extendable to settings where function approximation is required. Our approach is based on the successor representation (SR), which…
New algorithm reduces reinforcement learning complexity, approaching contextual bandits.
problem Episodic reinforcement learning's difficulty compared to contextual bandits.
method Proposes MVP algorithm with a new Bernstein-type bonus for episodic reinforcement learning.
result Achieves near-optimal regret bound of $O\left(\left(\sqrt{SAK} + S^2A
ight) \poly\log \left(SAHK
ight)
ight)$, improving state-of-the-art results.
Bayesian approach uses Gaussian process for reinforcement learning.
problem Robotic locomotion environments
method Bayesian actor-critic, model-free reinforcement learning with Gaussian process for exploration and policy optimization.
result Gaussian process method outperforms current algorithms in robotic locomotion environments.
New method boosts skill learning by encouraging optimistic exploration.
problem Intrinsic reward for exploration is inherently pessimistic.
method Derive an information gain auxiliary objective involving an ensemble of discriminators and rewarding policy disagreement.
result Improves skill learning in grid worlds and Atari games.
Diverse projection ensembles improve distributional reinforcement learning.
problem Learning the distribution of returns in reinforcement learning.
method Combining multiple projection methods to improve model diversity and exploration.
result Diverse projection ensembles lead to significant performance improvements in exploration tasks.
New algorithm achieves asymptotically optimal regret without horizon dependence.
problem Horizon-free regret minimization for reinforcement learning.
method Proposes a new algorithm and proves a regret upper bound.
result Regret upper bound of \(\tilde O(\sqrt{SAK} + S^8A^3)\) with failure probability \(\delta\).
Paper analyzes strategic underreporting in competitive insurance markets.
problem Strategic underreporting by insureds in competitive insurance markets.
method Develops a dynamic insurance market model with two competing companies and a continuum of insureds, examines the interaction between strategic underreporting and competitive pricing under a Bonus-Malus System framework.
result Establishes the existence and uniqueness of the insureds' optimal reporting barrier and its dependence on BMS premiums; proves the existence of Nash equilibrium premium strategies.
This work develops a unified framework for RLHF with general -divergence regularization.
problem Theoretical understanding of general -divergence regularization in RLHF.
method Holistic approach across -divergence class, two algorithms based on distinct sampling principles.
result Provably efficient algorithms with regret and sub-optimality gap.
Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcs.LG
XPO enhances RLHF by encouraging diverse responses, offering improved sample efficiency.
problem Limited exploration in RLHF leads to suboptimal models and bottlenecks.
method XPO is a simple one-line change to DPO, introducing a novel exploration bonus.
result XPO achieves strong theoretical guarantees and promising empirical performance.
Improved online Q-learning for MDPs with concentration bounds.
problem Online Q-learning in infinite-horizon discounted MDPs with sublinear regret for large gaps.
method Smoothed -Greedy exploration scheme combining -greedy and Boltzmann exploration, analyzed using concentration bounds for contractive Markovian stochastic approximation.
result Near- regret bound for Smoothed -Greedy exploration scheme.
In this work, we consider the popular tree-based search strategy within the framework of reinforcement learning, the Monte Carlo Tree Search (MCTS), in the context of infinite-horizon discounted cost Markov Decision Process (MDP). While MCTS is believed to provide an approximate value function for a given state with en…
This paper offers a financial economic perspective on the optimal time (and age) at which the owner of a Variable Annuity (VA) policy with a Guaranteed Living Withdrawal Benefit (GLWB) rider should initiate guaranteed lifetime income payments. We abstract from utility, bequest and consumption preference issues by treat…
UCB algorithm adapted for large-scale, non-sub-Gaussian problems.
problem Selecting the best alternative from a large set of options with non-sub-Gaussian performance distributions.
method Adapted UCB algorithm for non-sub-Gaussian settings, focusing on sample size and meta-UCB selection.
result UCB algorithms can achieve sample optimality in large-scale, non-sub-Gaussian problems.
Since the machine learning techniques are improving rapidly, it has been shown that the image recognition techniques in deep neural networks can be used to detect jet substructure. And it turns out that deep neural networks can match or outperform traditional approach of expert features. However, there are disadvantage…
New method improves RL in continuous spaces with kernel smoothing.
problem Sample efficiency and structural assumptions in classical RL.
method Kernel smoothing model-based approach with Bernstein-style exploration bonus.
result Achieves improved regret bound in finite-horizon settings.
Improves policy optimization with polylog(T) regret bounds for stochastic losses.
problem Improves theoretical guarantees for policy optimization in stochastic settings.
method Leverages Tsallis and Shannon entropy regularizers for polylog(T) regret, and log-barrier regularizer for adversarial settings.
result Achieves a first-order polylog(T) regret bound for policy optimization in stochastic settings.