On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,657 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “Preference-based Reinforcement Learning”

Provable Reward-Agnostic Preference-Based Reinforcement Learningcs.LG

Proposes a new theoretical framework for PbRL that requires less human feedback.

problem Lack of theoretical work capturing practical PbRL frameworks.

method Introduces a reward-agnostic PbRL framework that acquires exploratory trajectories before human feedback.

result Demonstrates improved sample complexity for learning optimal policies in linear and low-rank MDPs.

Preference-based Reinforcement Learning Reward-Agnostic Theoretical Analysis Human Feedback Sample Complexity

2023-05-29abs ↗pdf ↗

Dueling Posterior Sampling for Preference-Based Reinforcement Learningcs.LG

In preference-based reinforcement learning (RL), an agent interacts with the environment while receiving preferences instead of absolute feedback. While there is increasing research activity in preference-based RL, the design of formal frameworks that admit tractable theoretical analysis remains an open challenge. Buil…

2019-08-04abs ↗pdf ↗

Preference-based Reinforcement Learning with Finite-Time Guaranteescs.LG

Paper analyzes finite-time guarantees for preference-based RL.

problem Understanding finite-time guarantees for preference-based RL.

method Combines dueling bandits and policy search to navigate state space.

result Identifies best policy up to accuracy ε with high probability.

Reinforcement Learning Preference-based Finite-time Analysis Policy Search Dueling Bandits

2020-06-16abs ↗pdf ↗

SARA uses similarity to learn rewards robustly and adaptively.

problem Robustness to labeler errors and adaptability to diverse feedback formats.

method Contrastive framework that learns latent representations and computes rewards as similarities.

result Strong performance on offline RL benchmarks and diverse applications.

Reinforcement Learning Preference-based Robustness Contrastive Learning Latent Representation

2025-06-14abs ↗pdf ↗

Learning Kernel-Based MDPs from Episodic Preferential Feedbackstat.ML

Study preference-based reinforcement learning in episodic kernel MDPs.

problem Learning from episodic human preferences in reinforcement learning.

method Developed preference-based value estimation and confidence sets for kernel-based MDPs.

result Proved high-probability regret bounds that converge to optimal policy value.

reinforcement learning human feedback episodic MDPs kernel methods regret bounds

2026-05-22abs ↗pdf ↗

Provable Offline Preference-Based Reinforcement Learningcs.LG

Paper tackles offline preference-based RL with human feedback.

problem Offline Preference-based Reinforcement Learning with preference feedback.

method Two-step approach: MLE for reward estimation and distributionally robust planning.

result First guarantee for learning any target policy with polynomial samples.

Offline RL Preference-based Distributionally robust Maximum Likelihood Estimation Concentrability coefficient

2023-05-24abs ↗pdf ↗

Is RLHF More Difficult than Standard RL?cs.LG

Paper shows RLHF can be solved similarly to standard RL.

problem Difficulty of RLHF compared to standard RL.

method Reduction to reward-based RL techniques.

result RLHF can be solved using existing algorithms for reward-based RL.

Reinforcement Learning Human Feedback Reward-based RL Preference-based RL Robust RL

2023-06-25abs ↗pdf ↗

Unified Algorithms for RL with Decision-Estimation Coefficients: PAC, Reward-Free, Preference-Based Learning, and Beyondcs.LG

Unified algorithm tackles various RL goals like reward-free and preference-based learning.

problem Unified approach to multiple RL learning goals.

method Decision-Estimation Coefficient (DEC) framework.

result Unified algorithm handles various learning goals with a single framework.

Reinforcement Learning Unified Algorithms Decision-Estimation Coefficient Reward-Free Preference-Based Learning

2022-09-23abs ↗pdf ↗

Preference-based Reinforcement Learning beyond Pairwise Comparisons: Benefits of Multiple Optionscs.LG

Improved sample efficiency in preference-based RL with multiple comparisons.

problem Sample inefficiency in preference-based reinforcement learning with pairwise comparisons.

method Proposes M-AUPO, an algorithm that selects multiple actions by maximizing average uncertainty within subsets.

result Achieves a suboptimality gap of $O\left( \frac{d}{T} \sqrt{ \sum_{t=1}^T \frac{1}{|S_t|}} ight)$, improving performance with larger subsets.

Preference-based RL Multiple comparisons Sample efficiency Reinforcement learning Ranking feedback

2025-10-21abs ↗pdf ↗

Automatic design optimization of preference-based subjective evaluation with online learning in crowdsourcing environmentcs.HC

Optimizes crowdsourced preference-based subjective evaluation with online learning.

problem Large-scale evaluation of generative media using crowdsourcing due to combinatorial explosion.

method Automatic optimization of pair combination selections and evaluation volumes with online learning.

result Optimizes evaluation by reducing pair combinations and allocating optimal evaluation volumes.

crowdsourcing subjective evaluation online learning pair selection evaluation optimization

2024-03-10abs ↗pdf ↗

Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Designcs.LG

Meta-algorithm for efficient reinforcement learning from human preferences.

problem Learning from human preference comparisons in Markov decision processes.

method Randomized exploration and experimental design for batch comparison queries.

result Meta-algorithm achieves both regret and last-iterate guarantees with minimal preference queries.

reinforcement learning human feedback experimental design Markov decision processes randomized exploration

2025-06-11abs ↗pdf ↗

f-GRPO and Beyond: Divergence-Based Reinforcement Learning Algorithms for General LLM Alignmentcs.LG

Extends reinforcement learning alignment to scalar rewards, improving math reasoning.

problem Designing reinforcement learning algorithms for general LLM alignment.

method Introduces f-GRPO and f-HAL, estimating f-divergences between reward-aligned and unaligned distributions.

result Improves math reasoning RLVR tasks and mitigates reward hacking.

reinforcement learning alignment scalar rewards f-divergences math reasoning

2026-02-05abs ↗pdf ↗

Batch Active Preference-Based Learning of Reward Functionscs.LG

Data generation and labeling are usually an expensive part of learning for robotics. While active learning methods are commonly used to tackle the former problem, preference-based learning is a concept that attempts to solve the latter by querying users with preference questions. In this paper, we will develop a new al…

2018-10-10abs ↗pdf ↗

Kernelized Offline Contextual Dueling Banditscs.LG

Efficiently identifies good policies by choosing contexts for human feedback.

problem Efficiently identifying good policies in applications with high feedback costs.

method Introduces offline contextual dueling bandit setting and an upper-confidence-bound style algorithm.

result Proves a regret bound and shows superior performance over uniformly sampled contexts.

contextual dueling bandits human feedback reinforcement learning regret bound upper-confidence-bound

2023-07-21abs ↗pdf ↗

Privacy-Preserving Reinforcement Learning from Human Feedback via Decoupled Reward Modelingstat.ML

Privacy-preserving reinforcement learning from human feedback using decoupled reward modeling.

problem Training large language models with sensitive user information while preserving privacy.

method Proposes a privacy-preserving framework that imposes differential privacy on reward learning only.

result Privacy contributes an additional additive term to the suboptimality gap, and the upper bound is rate-optimal up to logarithmic factors.

privacy-preserving reinforcement learning human feedback differential privacy reward modeling

2026-03-23abs ↗pdf ↗

Outcome-Based Online Reinforcement Learning: Algorithms and Fundamental Limitscs.LG

Paper analyzes online reinforcement learning with outcome-based feedback, providing efficient algorithms and fundamental limits.

problem Assigning credit to actions in reinforcement learning with only endpoint rewards.

method Develops a provably sample-efficient algorithm for online reinforcement learning with general function approximation.

result Achieves

O(C_{ m cov} H^3/ε^2)

sample complexity, characterizing statistical separation between outcome-based and per-step rewards.

reinforcement learning outcome-based feedback sample complexity function approximation statistical efficiency

2025-05-26abs ↗pdf ↗

Imitating Language via Scalable Inverse Reinforcement Learningcs.LG

New approach uses inverse reinforcement learning to improve language model training.

problem Training large language models using imitation learning methods.

method Developed a new method of inverse reinforcement learning to optimize sequences directly.

result IRL-based fine-tuning leads to better performance and diversity in language generation.

language models reinforcement learning imitation learning inverse Q-learning temporal difference

2024-09-02abs ↗pdf ↗

Preference-based Conditional Treatment Effects and Policy Learningstat.ML

New framework estimates treatment effects based on preferences.

problem Estimating treatment effects with flexible outcomes.

method Preference-based Conditional Treatment Effect (CPTE) framework.

result CPTE provides interpretable targets and new identifiability conditions.

treatment effects flexible outcomes policy learning non-identifiability conditional probability

2026-02-03abs ↗pdf ↗

Batch Active Learning of Reward Functions from Human Preferencescs.LG

Efficiently learns reward functions with fewer queries and shorter computation times.

problem Expensive data generation and labeling in robot learning.

method Batch active preference-based learning methods using determinantal point processes (DPP) and heuristic alternatives.

result Our batch active learning algorithm requires only a few queries and computes them in a short amount of time.

robotics reward function active learning human preferences determinantal point processes

2024-02-24abs ↗pdf ↗

Meta-Router: Bridging Gold-standard and Preference-based Evaluations in Large Language Model Routingcs.LG

Meta-Router optimizes LLM selection using gold-standard and preference-based data.

problem Training a high-quality LLM router with combined data sources is challenging due to bias and scarcity.

method Developed an integrative causal router training framework to correct bias and improve routing accuracy.

result Our approach delivers more accurate routing and improves the trade-off between cost and quality.

LLM router causal inference gold-standard data preference-based data response evaluation

2025-09-29abs ↗pdf ↗

Uncovering the Source of Machine Biascs.LG

Study finds gender bias in human evaluators and shows how machine learning can mitigate it.

problem Gender bias in human decision-making on micro-lending platforms.

method Structural econometric model and machine learning algorithms trained on real-world data.

result Machine learning algorithms can mitigate both preference-based and belief-based biases.

gender bias micro-lending human evaluators machine learning welfare

2022-01-09abs ↗pdf ↗

Preference-Based Dynamic Ranking Structure Recognitionstat.ML

Paper introduces a novel framework for recognizing dynamic ranking structures in preference-based data.

problem Complex and noisy preference-based data often hide underlying homogeneous structures.

method Developed an approach to identify dynamic ranking groups using temporal penalties and spectral estimation. Introduced an objective function for detecting structural changes.

result Consistent recognition of ranking groups and structural changes in preference-based data.

ranking structure dynamic groups spectral estimation Bradley-Terry model structural changes

2025-09-29abs ↗pdf ↗

One-Shot Safety Alignment for Large Language Models via Optimal Dualizationcs.AI

A new method reduces the computational burden of safety alignment for large language models.

problem Safety concerns in large language models and the need to align them with human preferences.

method Optimal dualization approach to reduce constrained alignment to an unconstrained problem.

result Our algorithms MoCAN and PeCAN significantly reduce computational burden and improve training stability.

large language models safety alignment Reinforcement Learning from Human Feedback dualization convex optimization

2024-05-29abs ↗pdf ↗

Preference-based Online Learning with Dueling Bandits: A Surveycs.LG

In machine learning, the notion of multi-armed bandits refers to a class of online learning problems, in which an agent is supposed to simultaneously explore and exploit a given set of choice alternatives in the course of a sequential decision process. In the standard setting, the agent learns from stochastic feedback …

2018-07-30abs ↗pdf ↗

Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrationscs.LG

The performance of imitation learning is typically upper-bounded by the performance of the demonstrator. While recent empirical results demonstrate that ranked demonstrations allow for better-than-demonstrator performance, preferences over demonstrations may be difficult to obtain, and little is known theoretically abo…

2019-07-09abs ↗pdf ↗

From Parity to Preference-based Notions of Fairness in Classificationstat.ML

The adoption of automated, data-driven decision making in an ever expanding range of applications has raised concerns about its potential unfairness towards certain social groups. In this context, a number of recent studies have focused on defining, detecting, and removing unfairness from data-driven decision systems. …

2017-06-30abs ↗pdf ↗

Sample Complexity of Preference-Based Nonparametric Off-Policy Evaluation with Deep Networkscs.LG

This paper improves sample efficiency for off-policy evaluation with preference data.

problem Improving sample efficiency for off-policy evaluation with preference data.

method Using a deep neural network to learn the value function and leveraging manifold structure.

result Established a provably efficient guarantee for off-policy evaluation with RLHF.

sample complexity off-policy evaluation reinforcement learning human preference deep neural networks

2023-10-16abs ↗pdf ↗

Estimating Rationally Inattentive Utility Functions with Deep Clustering for Framing - Applications in YouTube Engagement Dynamicscs.LG

We consider a framework involving behavioral economics and machine learning. Rationally inattentive Bayesian agents make decisions based on their posterior distribution, utility function and information acquisition cost Renyi divergence which generalizes Shannon mutual information). By observing these decisions, how ca…

2018-12-23abs ↗pdf ↗

Understanding Negations in Information Processing: Learning from Replicating Human Behaviorcs.AI

Information systems experience an ever-growing volume of unstructured data, particularly in the form of textual materials. This represents a rich source of information from which one can create value for people, organizations and businesses. For instance, recommender systems can benefit from automatically understanding…

2017-04-18abs ↗pdf ↗

Preference-Based Batch and Sequential Teaching: Towards a Unified View of Modelscs.LG

Algorithmic machine teaching studies the interaction between a teacher and a learner where the teacher selects labeled examples aiming at teaching a target hypothesis. In a quest to lower teaching complexity and to achieve more natural teacher-learner interactions, several teaching models and complexity measures have b…

2019-10-24abs ↗pdf ↗

Task Transfer by Preference-Based Cost Learningcs.LG

The goal of task transfer in reinforcement learning is migrating the action policy of an agent to the target task from the source task. Given their successes on robotic action planning, current methods mostly rely on two requirements: exactly-relevant expert demonstrations or the explicitly-coded cost function on targe…

2018-05-12abs ↗pdf ↗

Avoiding

\mathbf{exp(R_{max})}

scaling in RLHF through Preference-based Explorationcs.LG

Improves RLHF sample efficiency by scaling reward complexity polynomially.

problem Exponential sample complexity in RLHF algorithms for skewed preferences.

method SE-POPO, an online RLHF algorithm that achieves polynomial sample complexity.

result SE-POPO outperforms existing algorithms in sample efficiency.

Reinforcement Learning Human Feedback Sample Efficiency Polynomial Complexity Skewed Preferences

2025-02-02abs ↗pdf ↗

Vector preference-based contextual bandits under distributional shiftscs.LG

A new policy for contextual bandits adapts to reward vector shifts.

problem Learning under reward vector shifts with ordered rewards.

method Adaptive-discretization and optimistic elimination policy.

result Established upper bounds on preference-based regret.

contextual bandits reward vector distribution shift regret analysis optimistic elimination

2025-08-21abs ↗pdf ↗

Preference-based Pure Explorationstat.ML

Study on identifying most preferred policy in bandits with vector-valued rewards.

problem Identifying the most preferred policy in bandits with vector-valued rewards.

method Derive a novel lower bound on sample complexity, design the Preference-based Track and Stop (PreTS) algorithm, and derive a new concentration inequality.

result The sample complexity of PreTS is asymptotically tight.

bandits vector-valued rewards preference cone Pareto optimal arms lower bound

2024-12-04abs ↗pdf ↗

Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problemscs.LG

This work compares human feedback methods for reward learning in bandits.

problem Understanding how human feedback affects the performance of reward learning methods.

method Theoretical comparison of human feedback approaches in offline contextual bandits.

result Human bias and uncertainty in feedback modeling impact the theoretical guarantees of reward learning methods.

bandits human feedback reward learning theoretical guarantees contextual bandits

2023-07-24abs ↗pdf ↗

When Does Trajectory-Level Supervision Permit Efficient Offline Reinforcement Learning?stat.ML

Trajectory-level supervision allows efficient offline reinforcement learning.

problem Offline reinforcement learning

method Developing a statistical theory for offline policy optimization from trajectory-level labels

result Proving a high-probability guarantee of order

\widetilde O(H^2\sqrt{C_{sa}(\pi^\star)/n})

offline RL trajectory-level supervision statistical theory optimization sample complexity

2026-06-16abs ↗pdf ↗

Enhancing Preference-based Linear Bandits via Human Response Timecs.LG

Enhances preference learning by incorporating response times into binary choices.

problem Limited information from binary choices about preference strength.

method Combines choices and response times using the EZ diffusion model.

result Response times improve utility estimation for strong preferences.

preference learning human response time bandits interactive systems utility estimation

2024-09-09abs ↗pdf ↗

Preference-Based Batch and Sequential Teachingcs.LG

Novel framework for teaching complexity in machine teaching models.

problem Understanding and comparing teaching models in batch and sequential settings.

method Developed a novel framework using preference functions to capture teaching complexity.

result Identified preference functions leading to linear teaching complexity in sequential models.

machine teaching teaching complexity batch models sequential models VC dimension

2020-10-17abs ↗pdf ↗

Reinforcement Learning from Adversarial Preferences in Tabular MDPscs.LG

New framework for reinforcement learning with adversarial preferences in tabular MDPs.

problem Learning from preferences rather than direct rewards in MDPs with adversarial settings.

method Developed PbMDPs framework, established lower bounds, and proposed algorithms for regret minimization.

result Achieved regret bounds of

Ω( (H^2 S K)^{1/3} T^{2/3} )

for PbMDPs with Borda scores.

Reinforcement Learning MDPs Adversarial Preferences Borda Scores Regret Minimization

2025-07-15abs ↗pdf ↗

Sample Efficient Preference Alignment in LLMs via Active Explorationcs.LG

Efficiently identifies good policies by choosing contexts for human feedback.

problem Efficiently acquiring human feedback for preference alignment in large language models.

method Formalizes active exploration as a dueling bandit problem and proposes an active exploration algorithm with a polynomial worst-case regret bound.

result Proposed method outperforms baselines with limited human preferences on various language models and datasets.

Preference Alignment Active Exploration Large Language Models Dueling Bandits Human Feedback

2023-12-01abs ↗pdf ↗

Learning Parametric Distributions from Samples and Preferencesstat.ML

Paper improves parameter estimation of continuous distributions using preference feedback.

problem Improving parameter estimation of continuous distributions.

method Preference-based M-estimators and deterministic preferences.

result Preference-based estimators achieve an estimation error scaling of O(1/n), significantly faster than sample-only methods.

parameter estimation continuous distributions preference feedback M-estimators asymptotic variance

2025-05-29abs ↗pdf ↗

Local Preferential Bayesian Optimizationcs.LG

Local PBO methods improve preferential BO in high-dimensional problems.

problem Efficiently optimizing preferential BO in high-dimensional settings.

method Adapting high-dimensional BO techniques to preferential feedback.

result Local PBO methods reduce cumulative regret compared to global baselines.

Bayesian optimization preferential feedback high-dimensional problems trust-region derivative-informed search

2026-06-01abs ↗pdf ↗

Sequential Preference-Based Optimizationcs.LG

Many real-world engineering problems rely on human preferences to guide their design and optimization. We present PrefOpt, an open source package to simplify sequential optimization tasks that incorporate human preference feedback. Our approach extends an existing latent variable model for binary preferences to allow f…

2018-01-09abs ↗pdf ↗

Demographic-Agnostic Fairness without Harmcs.LG

New algorithm ensures fairness without sacrificing accuracy.

problem Ensuring fairness in machine learning without harming accuracy.

method Demographic-Agnostic Fairness without Harm (DAFH) algorithm.

result DAFH algorithm achieves higher accuracy than existing methods.

fairness machine learning accuracy demographic harm

2025-09-28abs ↗pdf ↗

Deep Bayesian Reward Learning from Preferencescs.LG

Bayesian inverse reinforcement learning (IRL) methods are ideal for safe imitation learning, as they allow a learning agent to reason about reward uncertainty and the safety of a learned policy. However, Bayesian IRL is computationally intractable for high-dimensional problems because each sample from the posterior req…

2019-12-10abs ↗pdf ↗

FraPPE: Fast and Efficient Preference-based Pure Explorationcs.LG

FraPPE efficiently identifies Pareto optimal arms in multi-objective bandits.

problem Efficiently identifying Pareto optimal arms in multi-objective bandits with confidence.

method Deriving structural properties and using Frank-Wolfe optimisation to solve the maxmin optimisation problem.

result FraPPE achieves optimal sample complexity and identifies the exact Pareto set.

multi-objective bandits Pareto optimality Frank-Wolfe optimisation sample complexity

2025-08-22abs ↗pdf ↗

Mixture-Model Preference Learning for Many-Objective Bayesian Optimizationcs.LG

Bayesian framework learns latent preference archetypes for many-objective optimization.

problem Expanding space of trade-offs and context-dependent human values.

method Dirichlet-process mixture model for latent preference archetypes, hybrid queries for efficient information.

result Mixture-aware Bayesian optimization outperforms standard methods on synthetic and real-world benchmarks.

Bayesian optimization many-objective Dirichlet-process hybrid queries regret guarantee

2026-03-30abs ↗pdf ↗

Decomposition Strategies for Constructive Preference Elicitationstat.ML

We tackle the problem of constructive preference elicitation, that is the problem of learning user preferences over very large decision problems, involving a combinatorial space of possible outcomes. In this setting, the suggested configuration is synthesized on-the-fly by solving a constrained optimization problem, wh…

2017-11-22abs ↗pdf ↗