Paper shows RLHF can be solved similarly to standard RL.
problem Difficulty of RLHF compared to standard RL.
method Reduction to reward-based RL techniques.
result RLHF can be solved using existing algorithms for reward-based RL.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,738 papers · 148 categories
Trend · papers per month
48 results for “Reward-based Learning”
New method for LLMs to learn reasoning by optimizing latent variables.
problem Teaching LLMs to generate logical justifications for answers.
method Formalized reasoning as latent variable model, derived FEM objective, designed sampling schemes.
result Prompt Posterior Sampling (PPS) outperforms other schemes in learning to reason.
Platform combines RL and language models to study narrative influence on AI decisions.
problem Understanding how narrative elements shape AI decision-making.
method Dual-system architecture with reinforcement learning and language model integration.
result Initial experiments show narrative frameworks can influence AI decision-making.
Paper explores limits and possibilities of aligning LLMs with human preferences.
problem Aligning LLMs with diverse human preferences to ensure fairness and informed outcomes.
method Analysis of probabilistic representation of human preferences and preservation of diverse preferences.
result LLMs can't fully align with human preferences using reward-based approaches due to Condorcet cycles, but mixed strategies are statistically possible.
Learning about many things can provide numerous benefits to a reinforcement learning system. For example, learning many auxiliary value functions, in addition to optimizing the environmental reward, appears to improve both exploration and representation learning. The question we tackle in this paper is how to sculpt th…
We present a new algorithm that significantly improves the efficiency of exploration for deep Q-learning agents in dialogue systems. Our agents explore via Thompson sampling, drawing Monte Carlo samples from a Bayes-by-Backprop neural network. Our algorithm learns much faster than common exploration strategies such as …
The paper proposes a model reward scheme for collaborative ML based on Shapley value and information gain.
problem Designing fair incentives for collaborative machine learning.
method The paper proposes a reward scheme based on Shapley value and information gain, with properties like fairness and stability.
result The proposed reward scheme satisfies fairness and trade-offs between desirable properties via an adjustable parameter.
Despite the numerous advances, reinforcement learning remains away from widespread acceptance for autonomous controller design as compared to classical methods due to lack of ability to effectively tackle the reality gap. The reliance on absolute or deterministic reward as a metric for optimization process renders rein…
Study shows challenges in reinforcement learning math problems, proposing enhancements and a hardness measure.
problem Challenges in reinforcement learning finding rare high-reward instances.
method Combining combinatorial group theory, algorithmic enhancements, and topological hardness measure.
result Resolved mathematical questions and proposed enhancements for reinforcement learning.
This work shows synthetic gradients can outperform backpropagation in sample efficiency.
problem The efficiency of backpropagation in training neural networks.
method Unified vectorized feedback framework for loss-based and reward-based learning, introducing synthetic gradients.
result Synthetic gradients can achieve lower gradient-estimation mean squared error than backpropagation under certain conditions.
We propose a new reinforcement learning algorithm for partially observable Markov decision processes (POMDP) based on spectral decomposition methods. While spectral methods have been previously employed for consistent learning of (passive) latent variable models such as hidden Markov models, POMDPs are more challenging…
We propose a new reinforcement learning algorithm for partially observable Markov decision processes (POMDP) based on spectral decomposition methods. While spectral methods have been previously employed for consistent learning of (passive) latent variable models such as hidden Markov models, POMDPs are more challenging…
Combining model-based and model-free deep reinforcement learning has shown great promise for improving sample efficiency on complex control tasks while still retaining high performance. Incorporating imagination is a recent effort in this direction inspired by human mental simulation of motor behavior. We propose a lea…
DDO-RM improves reward-based policies by converting reward scores into a target distribution.
problem Improving reward-based policies when the reward function is simpler than the policy.
method Converts reward scores into a target distribution and uses KL-regularized mirror-descent updates.
result DDO-RM outperforms DPO in pair accuracy and mean margin.
Many currently deployed Reinforcement Learning agents work in an environment shared with humans, be them co-workers, users or clients. It is desirable that these agents adjust to people's preferences, learn faster thanks to their help, and act safely around them. We argue that most current approaches that learn from hu…
Improving Interaction Quality Estimation with BiLSTMs and the Impact on Dialogue Policy Learningcs.CL
The paper improves dialogue quality estimation using a novel user satisfaction model.
problem Improving dialogue quality estimation in spoken dialogue systems.
method Proposes a novel user satisfaction estimator based on BiLSTMs and reinforcement learning.
result The novel user satisfaction estimator outperforms previous models in terms of user satisfaction and task success.
Experience replay enables reinforcement learning agents to memorize and reuse past experiences, just as humans replay memories for the situation at hand. Contemporary off-policy algorithms either replay past experiences uniformly or utilize a rule-based replay strategy, which may be sub-optimal. In this work, we consid…
Paper introduces Latent-CLIP for efficient text-image comparison in latent space.
problem Efficiently compare text and images in latent space without costly decoding.
method Trains CLIP model in latent space, uses Latent-CLIP rewards for noise optimization, and guides generation away from harmful content.
result Latent-CLIP matches CLIP performance on text-image classification and harmful content detection.
New RLHF framework handles general preference oracles without reward functions.
problem Handling general preference oracles without assuming a reward function.
method Developed a minimax game between two LLMs for RLHF under a general preference oracle, focusing on KL-regularized preference.
result Proposed algorithms for efficient offline and online RLHF learning.
Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Designcs.LG
Meta-algorithm for efficient reinforcement learning from human preferences.
problem Learning from human preference comparisons in Markov decision processes.
method Randomized exploration and experimental design for batch comparison queries.
result Meta-algorithm achieves both regret and last-iterate guarantees with minimal preference queries.
Paper proposes a policy-search algorithm to learn entropy-maximizing exploration policies in reward-free environments.
problem Reward-free learning in high-dimensional, continuous-control domains.
method Maximum Entropy POLicy optimization (MEPOL) algorithm that maximizes a non-parametric state entropy estimate.
result MEPOL learns a maximum-entropy exploration policy that facilitates learning various reward-based tasks.
Study on indexability of restless multi-armed bandits and rollout policy performance.
problem Maximizing discounted rewards in finite state restless multi-armed bandit problems.
method Decouple the problem into single-armed restless bandits, analyze using value iteration, and compare with Whittle index policy.
result Demonstrates conditions for indexability and compares performance of index policy and rollout policy.
FinRLlama wins FinRL Challenge 2024 by fine-tuning LLMs with market data.
problem Lack of contextual alignment for financial market applications in traditional LLMs.
method Fine-tuning LLaMA-3.2-3B-Instruct model with custom RLMF prompt design integrating historical data and reward feedback.
result RLMF-tuned FinRLlama framework outperforms baseline methods in signal consistency and trading outcomes.
Heterogeneous Social Value Orientation Leads to Meaningful Diversity in Sequential Social Dilemmascs.MA
Heterogeneous SVO leads to diverse policies in sequential social dilemmas.
problem Understanding how diverse social value orientations affect behavior in sequential social dilemmas.
method Extending prior reinforcement learning studies, we instantiated heterogeneous SVO in a sequential social dilemma setting and measured task-specific diversity metrics.
result Heterogeneous SVO leads to meaningfully diverse policies across various incentive structures.
Novel framework for risk-sensitive reinforcement learning using martingale decomposition.
problem Risk sensitivity in sequential decision-making with uncertain rewards.
method Martingale decomposition and chaotic variation for reward uncertainty, integrated into model-free reinforcement learning algorithms.
result Demonstrated relevance of risk-sensitive reinforcement learning in grid world and portfolio optimization problems.
Recent reinforcement learning (RL) approaches have shown strong performance in complex domains such as Atari games, but are often highly sample inefficient. A common approach to reduce interaction time with the environment is to use reward shaping, which involves carefully designing reward functions that provide the ag…
ADVISOR dynamically balances imitation and reinforcement learning to overcome the imitation gap.
problem The gap between imitation learning and reinforcement learning when teaching agents have privileged information.
method Adaptive Insubordination (ADVISOR) dynamically weights imitation and reward-based reinforcement learning losses.
result On-the-fly switching with ADVISOR outperforms pure imitation, pure reinforcement learning, and their combinations.
The utility of learning a dynamics/world model of the environment in reinforcement learning has been shown in a many ways. When using neural networks, however, these models suffer catastrophic forgetting when learned in a lifelong or continual fashion. Current solutions to the continual learning problem require experie…
This paper balances short-term and long-term rewards in policy learning.
problem Balancing short-term and long-term rewards in policy learning.
method Formalizes a new framework to balance rewards, identifies rewards under mild assumptions, deduces efficiency bounds, and develops a policy learning approach.
result The proposed method improves the estimator of long-term reward and reduces regret.
This work improves RLHF performance guarantees using greedy sampling.
problem Learning the KL-regularized target with preference feedback.
method General preference model, greedy sampling.
result Order-wise improvements over existing RLHF performance guarantees.
The paper proposes a method to infer multi-objective rewards from preferences.
problem Modeling preferences based on multiple, often competing objectives.
method Modeling priorities lexicographically and inferring multi-objective rewards from observed preferences.
result Lexicographically-ordered rewards provide a better understanding of preferences and improve policies.
Model-based reinforcement learning (RL) is considered to be a promising approach to reduce the sample complexity that hinders model-free RL. However, the theoretical understanding of such methods has been rather limited. This paper introduces a novel algorithmic framework for designing and analyzing model-based RL algo…
New star-shaped acceptability indexes generalize existing methods.
problem Generalizing existing acceptability measures.
method Characterizing acceptability indexes through star-shaped risk measures and sets.
result Introducing concrete examples linked to various financial measures.
We study a variant of the stochastic multi-armed bandit (MAB) problem in which the rewards are corrupted. In this framework, motivated by privacy preservation in online recommender systems, the goal is to maximize the sum of the (unobserved) rewards, based on the observation of transformation of these rewards through a…
TROLL improves RL for LLMs by replacing clipping with a trust region projection.
problem Clipping in RL for LLMs causes instability and suboptimal performance.
method TROLL uses a discrete differentiable trust region projection to replace clipping, balancing computational cost and effectiveness.
result TROLL consistently outperforms PPO-like clipping in training speed, stability, and final success rates.
Efficient exploration remains a challenging problem in reinforcement learning, especially for those tasks where rewards from environments are sparse. A commonly used approach for exploring such environments is to introduce some "intrinsic" reward. In this work, we focus on model uncertainty estimation as an intrinsic r…
RLVR training dynamics reveal an implicit curriculum that shapes learning progression.
problem Understanding how RLVR overcomes the long-horizon barrier.
method Developed a theory of training dynamics for RLVR on transformers, using Fourier analysis on finite groups.
result Mixed-difficulty training naturally follows an implicit curriculum, shaping the learning progression from easy to hard.
ALINE optimizes Bayesian inference and data acquisition by strategically querying informative data.
problem Strategic acquisition of informative data for Bayesian inference in challenging tasks.
method Unified framework combining amortized Bayesian inference and active data acquisition using a transformer architecture trained via reinforcement learning.
result ALINE delivers both instant and accurate inference along with efficient selection of informative points.
In -armed bandit problem an agent sequentially interacts with environment which yields a reward based on the vector input the agent provides. The agent's goal is to maximise the sum of these rewards across some number of time steps. The problem and its variations have been a subject of numerous studies, su…
Reinforcement learning (RL) algorithms allow artificial agents to improve their selection of actions to increase rewarding experiences in their environments. Temporal Difference (TD) Learning -- a model-free RL method -- is a leading account of the midbrain dopamine system and the basal ganglia in reinforcement learnin…
RLVR learning dynamics naturally create an implicit curriculum for transformers.
problem How rewards based on final outcomes help overcome the long-horizon barrier in reasoning models.
method Developed a theory of training dynamics for RLVR on transformers, using Fourier analysis on finite groups.
result Mixed-difficulty training induces an implicit curriculum that shapes the learning progression from easy to hard.
Improves domain adaptation by combining multiple source domains and target domain data.
problem Poor performance of empirical risk minimization in distributionally shifted target domains.
method Distributionally robust model optimizing adversarial reward based on explained variance across multiple source domains.
result The robust model is a weighted average of conditional outcome models from source domains.
RLMH improves adaptive MCMC by optimizing contrastive divergence reward.
problem Tuning MCMC samplers is challenging and time-consuming.
method Formulated Metropolis-Hastings as a Markov decision process and used RL to adaptively tune it.
result A novel reward function based on contrastive divergence outperforms existing ones.
Zero-shot Policy Learning with Spatial Temporal RewardDecomposition on Contingency-aware Observationcs.LG
It is a long-standing challenge to enable an intelligent agent to learn in one environment and generalize to an unseen environment without further data collection and finetuning. In this paper, we consider a zero shot generalization problem setup that complies with biological intelligent agents' learning and generaliza…
A minimalist approach improves LLM reasoning by filtering incorrect responses.
problem Improving large language model (LLM) reasoning on complex tasks.
method Revisit GRPO from a reinforce-like algorithm perspective, proposing Reinforce-Rej.
result RAFT, a simple rejection sampling baseline, outperforms GRPO and PPO.
Satisficing is a relaxation of maximizing and allows for less risky decision making in the face of uncertainty. We propose two sets of satisficing objectives for the multi-armed bandit problem, where the objective is to achieve reward-based decision-making performance above a given threshold. We show that these new pro…
Autonomous cyber-physical agents and systems play an increasingly large role in our lives. To ensure that agents behave in ways aligned with the values of the societies in which they operate, we must develop techniques that allow these agents to not only maximize their reward in an environment, but also to learn and fo…
AGG-UCB uses neural networks to optimize group behaviors in contextual bandits.
problem Optimizing group behaviors in contextual bandits with mutual impacts.
method Introduces Arm Group Graph (AGG) and AGG-UCB algorithm using neural networks and graph neural networks.
result Achieves near-optimal regret bound with over-parameterized neural networks.