Paper tackles noisy reinforcement learning with perturbed rewards, improving agent performance.
problem Noisy rewards in reinforcement learning scenarios, leading to unreliable model performance.
method Develops a robust RL framework using a confusion matrix to estimate unbiased surrogate rewards.
result Trained policies using estimated surrogate rewards achieve higher expected rewards and faster convergence.
PHE adds pseudo-rewards to history to minimize regret in stochastic bandits.
problem Minimizing cumulative regret in stochastic multi-armed bandits.
method PHE algorithm that adds O ( t ) O(t) O ( t ) i.i.d. pseudo-rewards to history and pulls the best arm based on the perturbed history. result Near-optimal regret bounds derived for PHE.
EVILL uses randomised perturbations to improve exploration in bandit problems.
problem Improving exploration in structured stochastic bandit problems.
method Solves for the minimiser of a linearly perturbed regularised negative log-likelihood function.
result EVILL matches the performance of Thompson-sampling-style methods in theory and practice.
New algorithm minimizes regret in stochastic linear bandits with perturbed history.
problem Minimizing cumulative regret in stochastic linear bandits.
method Perturbed-history exploration in a linear bandit (LinPHE) algorithm.
result Achieves a O ( d n ) O(d \sqrt{n}) O ( d n ) gap-free bound on cumulative regret. Paper develops robust estimators and strategies for stochastic MABs with heavy-tailed rewards.
problem Stochastic multi-armed bandits with heavy-tailed rewards.
method Proposes a novel robust estimator and perturbation-based exploration strategy.
result Develops upper and lower regret bounds for various perturbations.
The paper shows how policy regularization acts like an adversary to improve robustness.
problem Improving robustness of learned policies in reinforcement learning.
method Using convex duality, the paper characterizes adversarial reward perturbations and provides generalization guarantees.
result Policy regularization acts as an adversary to improve robustness against worst-case reward perturbations.
New metrics improve scRNA-seq perturbation modeling by reducing mode collapse.
problem Outperformed by simple mean prediction in scRNA-seq perturbation modeling.
method Introduce DEG-aware metrics (WMSE, R w 2 ( Δ ) R^{2}_{w}(Δ) R w 2 ( Δ ) ) and negative/positive baselines. result WMSE loss function reduces mode collapse and improves model performance.
Unified analysis of perturbation-based strategies in stochastic and adversarial bandit problems.
problem Optimality of perturbation-based strategies in multi-armed bandit problems.
method Unified regret analysis for stochastic and adversarial settings, using perturbations of sub-Weibull and bounded support.
result Unified bounds for perturbations in both stochastic and adversarial settings, with optimal perturbations of Frechet-type.
Current reinforcement learning (RL) methods can successfully learn single tasks but often generalize poorly to modest perturbations in task domain or training procedure. In this work, we present a decoupled learning strategy for RL that creates a shared representation space where knowledge can be robustly transferred. …
This work explains RL policies using causal models, revealing important patterns and failures.
problem Understanding why RL policies succeed or fail in complex, high-dimensional systems.
method Developed a nonlinear Causal Model Reduction framework to learn simplified causal models from RL policy actions and rewards.
result The approach can uncover important behavioral patterns and failure modes in trained RL policies.
Robust Q-learning algorithm resists corrupted rewards.
problem Non-ideal environments with corrupted rewards.
method Developed a novel robust Q-learning algorithm using historical data.
result Finite-time convergence rate matches state-of-the-art bounds.
New method learns time-invariant rewards from demonstrations.
problem Learning robust rewards for tasks with varying execution times.
method Model-based inverse reinforcement learning with time-invariant costs.
result Approach enables learning from misaligned demonstrations and generalizes spatially.
A new exploration strategy for contextual bandits reduces regret and is computationally efficient.
problem Improving exploration in contextual bandits to reduce regret.
method Feature perturbation, injecting randomness directly into feature inputs.
result Achieves i l d e O ( d T ) ilde{\mathcal{O}}(d\sqrt{T}) i l d e O ( d T ) worst-case regret bound, surpassing existing methods. A new method boosts exploration in bandit algorithms, reducing regret.
problem Improving exploration in bandit algorithms with bounded or unbounded rewards.
method Residual Bootstrap Exploration (ReBoot) method that injects data-driven randomness.
result Proves logarithmic regret in Gaussian multi-armed bandits with appropriate variance inflation.
Reward-poisoning attacks can force RL agents to learn bad policies, and we categorize and quantify their feasibility.
problem Reward-poisoning attacks can manipulate RL agents to learn undesirable policies.
method Categorize attacks by infinity-norm constraint, provide thresholds for feasibility, and develop adaptive attack strategies.
result Adaptive reward-poisoning attacks can achieve the nefarious policy in polynomial steps, while non-adaptive attacks require exponential steps.
Reinforcement learning (RL) has advanced greatly in the past few years with the employment of effective deep neural networks (DNNs) on the policy networks. With the great effectiveness came serious vulnerability issues with DNNs that small adversarial perturbations on the input can change the output of the network. Sev…
New algorithm robust to probabilistic unbounded adversarial attacks in bandit problems.
problem Powerful adversaries that can catastrophically perturb the revealed reward in bandit problems.
method Proposes med-E-UCB and med- ε ε ε -greedy algorithms based on sample median for robustness. result Achieves O ( log T ) \mathcal{O}(\log T) O ( log T ) pseudo-regret under arbitrary and unbounded reward perturbation. ESPD improves learning efficiency in sparse reward reinforcement learning.
problem Sparse reward reinforcement learning challenges.
method Evolutionary Stochastic Policy Distillation (ESPD) based on drifted random walk insight.
result High learning efficiency demonstrated in MuJoCo robotics control suite experiments.
NADPEx uses dropout to enable temporally consistent exploration in reinforcement learning.
problem Achieving temporally consistent exploration in reinforcement learning agents.
method Integrates dropout into reinforcement learning policies to ensure temporal consistency.
result NADPEx outperforms naive exploration and parameter noise in tasks with sparse rewards.
Enhances RL in partially observable, noisy environments by uncovering causal states.
problem Making decisions based on incomplete and noisy observations in partially observable Markov decision processes (P 2 ^2 2 OMDPs). method Causal State Representation under Asynchronous Diffusion Model (CaDiff) framework, incorporating a novel asynchronous diffusion model (ADM) and a new bisimulation metric.
result Enhances returns by at least 14.18% compared to baselines on Roboschool tasks.
Attackers can significantly reduce team rewards in cooperative multi-agent reinforcement learning.
problem Robustness of cooperative multi-agent reinforcement learning to adversaries.
method Novel attack method involving training a policy network and using targeted adversarial examples.
result Reduces team reward from 20 to 9.4 by attacking a single agent, reducing winning rate from 98.9% to 0%.
We propose a framework for ensuring safe behavior of a reinforcement learning agent when the reward function may be difficult to specify. In order to do this, we rely on the existence of demonstrations from expert policies, and we provide a theoretical framework for the agent to optimize in the space of rewards consist…
MULEX separates exploration and exploitation in reinforcement learning.
problem Balancing discovery of new rewards with past behavior in reinforcement learning.
method Disentangles exploration and exploitation by optimizing multiple losses in parallel.
result MULEX achieves sample-efficiency and robustness in a hard-exploration environment.
State entropy regularization improves robustness in reinforcement learning, especially under structured perturbations.
problem Structured and spatially correlated perturbations in reinforcement learning.
method State entropy regularization, compared to policy entropy.
result State entropy regularization provides better robustness to structured and spatially correlated perturbations.
This work formalizes robustness criteria for reinforcement learning actions and improves performance in perturbed environments.
problem Improving reinforcement learning policies to perform well in uncertain or adversarial action scenarios.
method Formalized two robustness criteria for reinforcement learning actions, considering adversarial actions and action perturbations. Developed algorithms for tabular and deep reinforcement learning settings.
result Action-robust reinforcement learning policies improve performance in perturbed environments and are a form of implicit regularization.
Many practical environments contain catastrophic states that an optimal agent would visit infrequently or never. Even on toy problems, Deep Reinforcement Learning (DRL) agents tend to periodically revisit these states upon forgetting their existence under a new policy. We introduce intrinsic fear (IF), a learned reward…
This work develops a unified framework for RLHF with general f f f -divergence regularization.
problem Theoretical understanding of general f f f -divergence regularization in RLHF. method Holistic approach across f f f -divergence class, two algorithms based on distinct sampling principles. result Provably efficient algorithms with O ( log T ) O(\log T) O ( log T ) regret and O ( 1 / T ) O(1/T) O ( 1/ T ) sub-optimality gap. Proposes DRRO to mitigate over-optimization in RLHF from human feedback.
problem Over-optimization due to reward misspecification in RLHF.
method Wasserstein distributionally robust regret optimization (DRRO).
result DRRO mitigates over-optimization more effectively than existing baselines.
Optimal sample complexity analysis for plug-in approach in average-reward MDPs.
problem Learning optimal policies in average-reward MDPs with a generative model.
method Plug-in approach that constructs a model estimate and computes an optimal policy.
result Optimal sample complexities for the plug-in approach without prior knowledge of problem parameters.
A new recommendation system model tackles unreliable user behavior.
problem Creating effective recommendation systems in the presence of unreliable user behavior.
method A novel modification of Multi-Armed Bandits with an unreliable intermediate.
result Proved fundamental theorems and developed an Explore-Commit algorithm close to optimal performance.
Characterizes UCB algorithm performance in bandit problems.
problem Optimizing UCB algorithm performance in stochastic bandit environments.
method Novel perturbation analysis to characterize CLT of pulls and means.
result Smooth interpolation of pseudo-regret between large and small gap regimes.
Two randomized algorithms improve regret bounds for generalized linear bandits.
problem Improving regret bounds for generalized linear bandits.
method Two randomized algorithms: GLM-TSL and GLM-FPL.
result Upper bounds of O ( d n log K ) O(d \sqrt{n \log K}) O ( d n log K ) on regret for both algorithms. Algorithm mitigates performance loss in constrained reinforcement learning with model misspecification.
problem Performance loss in reinforcement learning policies due to model misspecification in constrained control systems.
method Proposes an algorithm to handle constrained model misspecification in continuous control systems.
result Algorithm successfully mitigates performance loss in real-world reinforcement learning tasks.
Two randomized algorithms improve performance in non-stationary linear bandits.
problem Conservatism in optimistic algorithms for non-stationary linear bandits.
method Two perturbation approaches: randomization and random perturbations.
result D-RandLinUCB and D-LinTS achieve optimal dynamic regret and are oracle-efficient.
The paper provides robustness guarantees for mode estimation in bandits.
problem Understanding robustness in mode estimation under adversarial data contamination.
method Simple randomization and theoretical analysis of multi-armed bandits.
result Regret guarantees for various modal bandit problems.
In this paper, we present an online reinforcement learning algorithm, called Renewal Monte Carlo (RMC), for infinite horizon Markov decision processes with a designated start state. RMC is a Monte Carlo algorithm and retains the advantages of Monte Carlo methods including low bias, simplicity, and ease of implementatio…
In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in rewards in addition to maximizing a standard criterion. Variance related risk measures are among the most common risk-sensitive criteria in finance and operations research. However, optimizing many such c…
New algorithm optimizes multi-armed bandits with low computational cost.
problem Optimizing multi-armed bandits with low computational cost.
method Proposes a new FTPL algorithm with optimistic principle for ambiguity.
result Unified regret analysis and low computational costs.
RADIAL-RL improves deep RL agents' robustness against adversarial attacks.
problem Vulnerability of deep reinforcement learning agents to small adversarial perturbations.
method RADIAL-RL, a principled framework for training robust reinforcement learning agents.
result RADIAL-RL-trained agents consistently outperform prior methods in robustness tests.
Polynomial chaos surrogates quantify epistemic uncertainty in AI-driven scientific models.
problem Uncertainty in reward estimates hinders interpretability in sequential generative models.
method Fit polynomial chaos expansions to trained models to propagate epistemic uncertainty and quantify sensitivity.
result Interpretable decomposition of reward components driving generative decisions.
ZDPG learns model-free policies without critics, improving on PG.
problem Model-free policy learning in complex dynamic problems.
method Approximates policy-reward gradients via two-point stochastic evaluations of the Q-function.
result Restores true model-free policy learning without critics, with improved stability and efficiency.
ACP-UCB1 ranks arms based on upper-tail performance, improving stochastic bandit algorithms.
problem Stochastic bandit algorithms often favor arms with strong upper-tail performance, which is not well-addressed by classical mean-reward criteria.
method ACP-UCB1 combines an adaptive conformal estimate of the upper endpoint with a UCB-type optimism bonus.
result ACP-UCB1 achieves logarithmic upper-quantile regret with per-arm contribution \(O(
icefrac{\log n}{Δ_j^{\mathrm{ACP}}})\).
Reward hacking exploits misspecified rewards, affecting agent capabilities and true performance.
problem Reward hacking in RL models exploiting reward misspecifications.
method Constructed four RL environments with misspecified rewards; analyzed agent capabilities and behavior.
result More capable agents exploit reward misspecifications, achieving higher proxy reward but lower true reward.
Paper introduces PRMs to learn non-Markovian stochastic rewards for reinforcement learning.
problem Lack of structured representation for non-Markovian stochastic rewards in reinforcement learning.
method Introduces probabilistic reward machines (PRMs) and presents an algorithm to learn them from decision processes.
result Algorithm proves correct and convergent for learning PRMs from decision processes.
Paper addresses reward learning issues in RL, improving both under- and over-estimation.
problem Reward learning from data can lead to reward delusions or underestimation, causing unintended behaviors.
method Connects reward learning to positive-unlabeled (PU) learning and applies a large-scale PU learning algorithm.
result Improves both GAIL and supervised reward learning without additional assumptions.
This work analyzes the value of future reward information in RL.
problem Analyzing the impact of knowing future rewards in reinforcement learning.
method Competitive analysis and worst-case reward distribution.
result Exact ratios between standard RL agents and those with future-reward lookahead.
Method learns reward functions that are independently obtainable and sum to original reward.
problem Learning reward functions that are independent and meaningful.
method Defining independent obtainability and optimizing a novel objective function.
result Learned reward functions generalize well to modified environments and have optimal policies.
Self-supervised reward prediction improves RL in sparse reward settings.
problem Data efficiency and sparse reward signals in reinforcement learning.
method Learning a state representation for reward prediction and using it to shape rewards.
result Self-supervised reward prediction enhances RL algorithms in single-goal environments.