Paper shows RLHF can be solved similarly to standard RL.
problem Difficulty of RLHF compared to standard RL.
method Reduction to reward-based RL techniques.
result RLHF can be solved using existing algorithms for reward-based RL.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Reward-based RL”
Platform combines RL and language models to study narrative influence on AI decisions.
problem Understanding how narrative elements shape AI decision-making.
method Dual-system architecture with reinforcement learning and language model integration.
result Initial experiments show narrative frameworks can influence AI decision-making.
Model-based reinforcement learning (RL) is considered to be a promising approach to reduce the sample complexity that hinders model-free RL. However, the theoretical understanding of such methods has been rather limited. This paper introduces a novel algorithmic framework for designing and analyzing model-based RL algo…
Modeling decision-making dynamics in MDD patients using RL-HMM.
problem Characterize reward learning strategies in MDD patients.
method Proposed RL-HMM framework to analyze reward-based decision-making.
result MDD patients show reduced engagement in RL compared to healthy controls.
Recent reinforcement learning (RL) approaches have shown strong performance in complex domains such as Atari games, but are often highly sample inefficient. A common approach to reduce interaction time with the environment is to use reward shaping, which involves carefully designing reward functions that provide the ag…
TROLL improves RL for LLMs by replacing clipping with a trust region projection.
problem Clipping in RL for LLMs causes instability and suboptimal performance.
method TROLL uses a discrete differentiable trust region projection to replace clipping, balancing computational cost and effectiveness.
result TROLL consistently outperforms PPO-like clipping in training speed, stability, and final success rates.
New method for LLMs to learn reasoning by optimizing latent variables.
problem Teaching LLMs to generate logical justifications for answers.
method Formalized reasoning as latent variable model, derived FEM objective, designed sampling schemes.
result Prompt Posterior Sampling (PPS) outperforms other schemes in learning to reason.
A minimalist approach improves LLM reasoning by filtering incorrect responses.
problem Improving large language model (LLM) reasoning on complex tasks.
method Revisit GRPO from a reinforce-like algorithm perspective, proposing Reinforce-Rej.
result RAFT, a simple rejection sampling baseline, outperforms GRPO and PPO.
RLMH improves adaptive MCMC by optimizing contrastive divergence reward.
problem Tuning MCMC samplers is challenging and time-consuming.
method Formulated Metropolis-Hastings as a Markov decision process and used RL to adaptively tune it.
result A novel reward function based on contrastive divergence outperforms existing ones.
Reinforcement Learning based Sequential Batch-sampling for Bayesian Optimal Experimental Designcs.LG
This work uses RL to optimize batch experiments in SDOE.
problem Maximizing knowledge with limited resources and constraints.
method Reinforcement Learning for batch-sampling in Bayesian SDOE.
result The proposed algorithm optimizes experiment selection for multiple tasks.
Offline Reinforcement Learning for Human-Guided Human-Machine Interaction with Private Informationstat.ML
Study human-machine interaction with private info using offline RL.
problem Confounding bias and distributional mismatch in offline RL for human-guided interaction.
method Developed a novel identification result and OPE method to address confounding bias, and used pessimism to tackle distributional mismatch.
result Policy pair converges to optimal one at satisfactory rate under mild assumptions.
Reinforcement learning (RL) algorithms allow artificial agents to improve their selection of actions to increase rewarding experiences in their environments. Temporal Difference (TD) Learning -- a model-free RL method -- is a leading account of the midbrain dopamine system and the basal ganglia in reinforcement learnin…
This work improves RLHF performance guarantees using greedy sampling.
problem Learning the KL-regularized target with preference feedback.
method General preference model, greedy sampling.
result Order-wise improvements over existing RLHF performance guarantees.
Paper explores limits and possibilities of aligning LLMs with human preferences.
problem Aligning LLMs with diverse human preferences to ensure fairness and informed outcomes.
method Analysis of probabilistic representation of human preferences and preservation of diverse preferences.
result LLMs can't fully align with human preferences using reward-based approaches due to Condorcet cycles, but mixed strategies are statistically possible.
RLVR training dynamics reveal an implicit curriculum that shapes learning progression.
problem Understanding how RLVR overcomes the long-horizon barrier.
method Developed a theory of training dynamics for RLVR on transformers, using Fourier analysis on finite groups.
result Mixed-difficulty training naturally follows an implicit curriculum, shaping the learning progression from easy to hard.
One crucial aspect of partial domain adaptation (PDA) is how to select the relevant source samples in the shared classes for knowledge transfer. Previous PDA methods tackle this problem by re-weighting the source samples based on their high-level information (deep features). However, since the domain shift between sour…
Zero-shot Policy Learning with Spatial Temporal RewardDecomposition on Contingency-aware Observationcs.LG
It is a long-standing challenge to enable an intelligent agent to learn in one environment and generalize to an unseen environment without further data collection and finetuning. In this paper, we consider a zero shot generalization problem setup that complies with biological intelligent agents' learning and generaliza…
The paper proposes a model reward scheme for collaborative ML based on Shapley value and information gain.
problem Designing fair incentives for collaborative machine learning.
method The paper proposes a reward scheme based on Shapley value and information gain, with properties like fairness and stability.
result The proposed reward scheme satisfies fairness and trade-offs between desirable properties via an adjustable parameter.
We present a new algorithm that significantly improves the efficiency of exploration for deep Q-learning agents in dialogue systems. Our agents explore via Thompson sampling, drawing Monte Carlo samples from a Bayes-by-Backprop neural network. Our algorithm learns much faster than common exploration strategies such as …
Paper introduces Latent-CLIP for efficient text-image comparison in latent space.
problem Efficiently compare text and images in latent space without costly decoding.
method Trains CLIP model in latent space, uses Latent-CLIP rewards for noise optimization, and guides generation away from harmful content.
result Latent-CLIP matches CLIP performance on text-image classification and harmful content detection.
Learning about many things can provide numerous benefits to a reinforcement learning system. For example, learning many auxiliary value functions, in addition to optimizing the environmental reward, appears to improve both exploration and representation learning. The question we tackle in this paper is how to sculpt th…
Study on indexability of restless multi-armed bandits and rollout policy performance.
problem Maximizing discounted rewards in finite state restless multi-armed bandit problems.
method Decouple the problem into single-armed restless bandits, analyze using value iteration, and compare with Whittle index policy.
result Demonstrates conditions for indexability and compares performance of index policy and rollout policy.
Interactive RL and DT feedback improve feature selection efficiency.
problem Balancing feature selection effectiveness and efficiency.
method Interactive Reinforcement Learning (IRL) and Decision Tree Feedback (DTF) architecture.
result Improved feature selection performance on real-world datasets.
Hybrid RL algorithms improve offline and online RL in linear MDPs.
problem Improving RL performance without single-policy concentrability.
method Developed computationally efficient algorithms for PAC and regret-minimizing RL in linear MDPs.
result Achieved sharper error or regret bounds for linear MDPs.
This work explores representation complexity in RL paradigms, revealing model-based RL as the easiest task.
problem Investigating the representation complexity gap among model-based, policy-based, and value-based RL.
method Demonstrated through analysis of Markov decision processes (MDPs) and introduced new classes of MDPs.
result Representation complexity hierarchy: model-based RL > policy-based RL > value-based RL.
New method defends RL agents from poisoning attacks without MDP knowledge.
problem Poisoning attacks on RL systems can cause learning failures.
method Generic poisoning framework for online RL, Vulnerability-Aware Adversarial Critic Poison (VA2C-P).
result Successfully prevents RL agents from learning good policies or converging to target policies.
Reincarnating RL reuses prior work to accelerate RL progress.
problem Efficiency and accessibility in reinforcement learning for large-scale applications.
method Transfer of learned policies between RL agents or design iterations, focusing on value-based RL.
result Demonstrated gains in performance over tabula rasa RL on various tasks.
RL tackles decision making in unknown environments, focusing on efficiency and efficacy.
problem Efficiency and efficacy in RL algorithms for sample-starved situations.
method Markov Decision Processes, model-based and value-based approaches, policy optimization.
result Enhanced understanding and improvements in sample and computational efficacies of RL algorithms.
Catalyst.RL accelerates RL research with efficient training.
problem Efficient reinforcement learning training in complex environments.
method Open-source PyTorch framework with distributed training and RL algorithms.
result Catalyst.RL achieved 2nd place in a computationally expensive RL challenge.
Study shows effectiveness of offline RL in online RL tasks.
problem Improving online RL efficiency using offline RL data.
method Formalized framework for incorporating offline RL as online RL subroutines, introducing techniques to enhance effectiveness.
result Effectiveness of the framework depends on task nature, techniques greatly enhance effectiveness, and existing methods are ineffective.
MOReL learns offline RL policies using pessimistic MDPs.
problem Offline RL's data efficiency and velocity.
method Two-step process: learn P-MDP and near-optimal policy in it.
result MOReL is minimax optimal and matches state-of-the-art results.
New star-shaped acceptability indexes generalize existing methods.
problem Generalizing existing acceptability measures.
method Characterizing acceptability indexes through star-shaped risk measures and sets.
result Introducing concrete examples linked to various financial measures.
We study a variant of the stochastic multi-armed bandit (MAB) problem in which the rewards are corrupted. In this framework, motivated by privacy preservation in online recommender systems, the goal is to maximize the sum of the (unobserved) rewards, based on the observation of transformation of these rewards through a…
RL research overhypes potential but lacks deployable solutions.
problem Current RL research overhypes potential and lacks deployable solutions.
method Identifies and critiques current RL research practices.
result Current RL research direction is unlikely to lead to practical, economically viable solutions.
Despite the numerous advances, reinforcement learning remains away from widespread acceptance for autonomous controller design as compared to classical methods due to lack of ability to effectively tackle the reality gap. The reliance on absolute or deterministic reward as a metric for optimization process renders rein…
RL applied to finance tasks, highlighting challenges and future directions.
problem Decision-making tasks in finance using RL.
method Meta-analysis of RL applications, identifying challenges and proposing future directions.
result Challenges in RL performance and future research directions.
Reinforcement learning (RL) algorithms allow agents to learn skills and strategies to perform complex tasks without detailed instructions or expensive labelled training examples. That is, RL agents can learn, as we learn. Given the importance of learning in our intelligence, RL has been thought to be one of key compone…
A simple approach to offline RL without additional complexity.
problem Learning from a fixed dataset of actions with value estimation errors.
method Adding a behavior cloning term to the policy update of an online RL algorithm and normalizing the data.
result Matches the performance of state-of-the-art offline RL algorithms with minimal changes.
MiniHack simplifies creation of complex RL environments.
problem Limited availability of challenging RL benchmarks.
method Develops a sandbox framework for easy RL environment design.
result MiniHack enables rapid creation of diverse RL testbeds.
Generative flow networks use RL to learn probabilistic models efficiently.
problem Training generative models with RL for compositional discrete objects.
method Reformulate GFlowNet training as entropy-regularized RL with specific reward and regularizer.
result Entropy-regularized RL can be competitive with established GFlowNet training methods.
Experimentally, it has been observed that humans and animals often make decisions that do not maximize their expected utility, but rather choose outcomes randomly, with probability proportional to expected utility. Probability matching, as this strategy is called, is equivalent to maximum entropy reinforcement learning…
RL agents outperform baselines in asset allocation.
problem Optimizing asset allocation using reinforcement learning.
method Model-free deep RL agents trained on real-world stock prices.
result RL agents significantly outperformed random and uniform allocation.
Study shows challenges in reinforcement learning math problems, proposing enhancements and a hardness measure.
problem Challenges in reinforcement learning finding rare high-reward instances.
method Combining combinatorial group theory, algorithmic enhancements, and topological hardness measure.
result Resolved mathematical questions and proposed enhancements for reinforcement learning.
This paper shows RL with KL penalties is equivalent to Bayesian inference for fine-tuning LMs.
problem Fine-tuning large language models to avoid undesirable features.
method Analyzed KL-regularized RL and showed it's equivalent to variational inference.
result KL-regularized RL avoids distribution collapse and is more insightful as Bayesian inference.
Despite the recent progress in deep reinforcement learning field (RL), and, arguably because of it, a large body of work remains to be done in reproducing and carefully comparing different RL algorithms. We present catalyst.RL, an open source framework for RL research with a focus on reproducibility and flexibility. Ma…
Improving Interaction Quality Estimation with BiLSTMs and the Impact on Dialogue Policy Learningcs.CL
Learning suitable and well-performing dialogue behaviour in statistical spoken dialogue systems has been in the focus of research for many years. While most work which is based on reinforcement learning employs an objective measure like task success for modelling the reward signal, we use a reward based on user satisfa…
Since their introduction a year ago, distributional approaches to reinforcement learning (distributional RL) have produced strong results relative to the standard approach which models expected values (expected RL). However, aside from convergence guarantees, there have been few theoretical results investigating the re…
HTMRL uses HTM for RL, adapting faster to changing environments.
problem Adapting to non-stationary environments in RL.
method Strictly HTM-based RL algorithm.
result HTMRL adapts faster to changing environments in a 10-armed bandit.