Meta-Router optimizes LLM selection using gold-standard and preference-based data.
problem Training a high-quality LLM router with combined data sources is challenging due to bias and scarcity.
method Developed an integrative causal router training framework to correct bias and improve routing accuracy.
result Our approach delivers more accurate routing and improves the trade-off between cost and quality.
Paper introduces a novel framework for recognizing dynamic ranking structures in preference-based data.
problem Complex and noisy preference-based data often hide underlying homogeneous structures.
method Developed an approach to identify dynamic ranking groups using temporal penalties and spectral estimation. Introduced an objective function for detecting structural changes.
result Consistent recognition of ranking groups and structural changes in preference-based data.
Optimizes crowdsourced preference-based subjective evaluation with online learning.
problem Large-scale evaluation of generative media using crowdsourcing due to combinatorial explosion.
method Automatic optimization of pair combination selections and evaluation volumes with online learning.
result Optimizes evaluation by reducing pair combinations and allocating optimal evaluation volumes.
In preference-based reinforcement learning (RL), an agent interacts with the environment while receiving preferences instead of absolute feedback. While there is increasing research activity in preference-based RL, the design of formal frameworks that admit tractable theoretical analysis remains an open challenge. Buil…
The adoption of automated, data-driven decision making in an ever expanding range of applications has raised concerns about its potential unfairness towards certain social groups. In this context, a number of recent studies have focused on defining, detecting, and removing unfairness from data-driven decision systems. …
Data generation and labeling are usually an expensive part of learning for robotics. While active learning methods are commonly used to tackle the former problem, preference-based learning is a concept that attempts to solve the latter by querying users with preference questions. In this paper, we will develop a new al…
Paper tackles offline preference-based RL with human feedback.
problem Offline Preference-based Reinforcement Learning with preference feedback.
method Two-step approach: MLE for reward estimation and distributionally robust planning.
result First guarantee for learning any target policy with polynomial samples.
Proposes a new theoretical framework for PbRL that requires less human feedback.
problem Lack of theoretical work capturing practical PbRL frameworks.
method Introduces a reward-agnostic PbRL framework that acquires exploratory trajectories before human feedback.
result Demonstrates improved sample complexity for learning optimal policies in linear and low-rank MDPs.
Efficiently learns reward functions with fewer queries and shorter computation times.
problem Expensive data generation and labeling in robot learning.
method Batch active preference-based learning methods using determinantal point processes (DPP) and heuristic alternatives.
result Our batch active learning algorithm requires only a few queries and computes them in a short amount of time.
Study finds gender bias in human evaluators and shows how machine learning can mitigate it.
problem Gender bias in human decision-making on micro-lending platforms.
method Structural econometric model and machine learning algorithms trained on real-world data.
result Machine learning algorithms can mitigate both preference-based and belief-based biases.
Paper analyzes finite-time guarantees for preference-based RL.
problem Understanding finite-time guarantees for preference-based RL.
method Combines dueling bandits and policy search to navigate state space.
result Identifies best policy up to accuracy ε with high probability.
New framework estimates treatment effects based on preferences.
problem Estimating treatment effects with flexible outcomes.
method Preference-based Conditional Treatment Effect (CPTE) framework.
result CPTE provides interpretable targets and new identifiability conditions.
In machine learning, the notion of multi-armed bandits refers to a class of online learning problems, in which an agent is supposed to simultaneously explore and exploit a given set of choice alternatives in the course of a sequential decision process. In the standard setting, the agent learns from stochastic feedback …
Algorithmic machine teaching studies the interaction between a teacher and a learner where the teacher selects labeled examples aiming at teaching a target hypothesis. In a quest to lower teaching complexity and to achieve more natural teacher-learner interactions, several teaching models and complexity measures have b…
SARA uses similarity to learn rewards robustly and adaptively.
problem Robustness to labeler errors and adaptability to diverse feedback formats.
method Contrastive framework that learns latent representations and computes rewards as similarities.
result Strong performance on offline RL benchmarks and diverse applications.
Study on identifying most preferred policy in bandits with vector-valued rewards.
problem Identifying the most preferred policy in bandits with vector-valued rewards.
method Derive a novel lower bound on sample complexity, design the Preference-based Track and Stop (PreTS) algorithm, and derive a new concentration inequality.
result The sample complexity of PreTS is asymptotically tight.
Unified algorithm tackles various RL goals like reward-free and preference-based learning.
problem Unified approach to multiple RL learning goals.
method Decision-Estimation Coefficient (DEC) framework.
result Unified algorithm handles various learning goals with a single framework.
Improved sample efficiency in preference-based RL with multiple comparisons.
problem Sample inefficiency in preference-based reinforcement learning with pairwise comparisons.
method Proposes M-AUPO, an algorithm that selects multiple actions by maximizing average uncertainty within subsets.
result Achieves a suboptimality gap of $O\left( \frac{d}{T} \sqrt{ \sum_{t=1}^T \frac{1}{|S_t|}}
ight)$, improving performance with larger subsets.
Paper shows RLHF can be solved similarly to standard RL.
problem Difficulty of RLHF compared to standard RL.
method Reduction to reward-based RL techniques.
result RLHF can be solved using existing algorithms for reward-based RL.
Novel framework for teaching complexity in machine teaching models.
problem Understanding and comparing teaching models in batch and sequential settings.
method Developed a novel framework using preference functions to capture teaching complexity.
result Identified preference functions leading to linear teaching complexity in sequential models.
A new policy for contextual bandits adapts to reward vector shifts.
problem Learning under reward vector shifts with ordered rewards.
method Adaptive-discretization and optimistic elimination policy.
result Established upper bounds on preference-based regret.
This work compares human feedback methods for reward learning in bandits.
problem Understanding how human feedback affects the performance of reward learning methods.
method Theoretical comparison of human feedback approaches in offline contextual bandits.
result Human bias and uncertainty in feedback modeling impact the theoretical guarantees of reward learning methods.
Many real-world engineering problems rely on human preferences to guide their design and optimization. We present PrefOpt, an open source package to simplify sequential optimization tasks that incorporate human preference feedback. Our approach extends an existing latent variable model for binary preferences to allow f…
Study preference-based reinforcement learning in episodic kernel MDPs.
problem Learning from episodic human preferences in reinforcement learning.
method Developed preference-based value estimation and confidence sets for kernel-based MDPs.
result Proved high-probability regret bounds that converge to optimal policy value.
Enhances preference learning by incorporating response times into binary choices.
problem Limited information from binary choices about preference strength.
method Combines choices and response times using the EZ diffusion model.
result Response times improve utility estimation for strong preferences.
Paper improves parameter estimation of continuous distributions using preference feedback.
problem Improving parameter estimation of continuous distributions.
method Preference-based M-estimators and deterministic preferences.
result Preference-based estimators achieve an estimation error scaling of O(1/n), significantly faster than sample-only methods.
Local PBO methods improve preferential BO in high-dimensional problems.
problem Efficiently optimizing preferential BO in high-dimensional settings.
method Adapting high-dimensional BO techniques to preferential feedback.
result Local PBO methods reduce cumulative regret compared to global baselines.
New algorithm ensures fairness without sacrificing accuracy.
problem Ensuring fairness in machine learning without harming accuracy.
method Demographic-Agnostic Fairness without Harm (DAFH) algorithm.
result DAFH algorithm achieves higher accuracy than existing methods.
FraPPE efficiently identifies Pareto optimal arms in multi-objective bandits.
problem Efficiently identifying Pareto optimal arms in multi-objective bandits with confidence.
method Deriving structural properties and using Frank-Wolfe optimisation to solve the maxmin optimisation problem.
result FraPPE achieves optimal sample complexity and identifies the exact Pareto set.
Study creates web interface to elicit user-preferred metrics.
problem Eliciting classification metrics that align with user preferences.
method Developed a web-based interface and conducted a user study.
result Users preferred metrics that align with their task and context.
We consider a framework involving behavioral economics and machine learning. Rationally inattentive Bayesian agents make decisions based on their posterior distribution, utility function and information acquisition cost Renyi divergence which generalizes Shannon mutual information). By observing these decisions, how ca…
Information systems experience an ever-growing volume of unstructured data, particularly in the form of textual materials. This represents a rich source of information from which one can create value for people, organizations and businesses. For instance, recommender systems can benefit from automatically understanding…
With ever-increasing available data, predicting individuals' preferences and helping them locate the most relevant information has become a pressing need. Understanding and predicting preferences is also important from a fundamental point of view, as part of what has been called a "new" computational social science. He…
Efficiently identifies good policies by choosing contexts for human feedback.
problem Efficiently identifying good policies in applications with high feedback costs.
method Introduces offline contextual dueling bandit setting and an upper-confidence-bound style algorithm.
result Proves a regret bound and shows superior performance over uniformly sampled contexts.
This paper develops a new method for eliciting more flexible metrics, improving fairness and applicability.
problem Limited flexibility in existing metric elicitation strategies for reflecting user preferences.
method Develops a strategy for eliciting quadratic metrics based on predictive rates, requiring only relative preference feedback.
result Achieves near-optimal query complexity and broadens the use cases for metric elicitation.
Proposes auditing for envy-freeness in recommender systems to assess individual preferences.
problem Auditing fairness in recommender systems for individual preferences.
method Formulates a pure exploration problem in multi-armed bandits, proposing a sample-efficient algorithm with theoretical guarantees.
result Algorithm ensures fairness without deteriorating user experience on real-world datasets.
LLM safety alignment explained as divergence estimation.
problem Aligning large language models to avoid harmful outputs.
method Presented a theoretical framework showing alignment methods as divergence estimators.
result KLDO method improves safety alignment using compliance-refusal datasets.
New algorithms improve dueling bandit performance in multiplayer settings.
problem Challenges in collaborative exploration of non-informative arm pairs in multiplayer dueling bandits.
method Demonstrated Follow Your Leader approach and message-passing fully distributed protocol.
result Multiplayer algorithms outperform single-player benchmarks.
Bayesian framework learns latent preference archetypes for many-objective optimization.
problem Expanding space of trade-offs and context-dependent human values.
method Dirichlet-process mixture model for latent preference archetypes, hybrid queries for efficient information.
result Mixture-aware Bayesian optimization outperforms standard methods on synthetic and real-world benchmarks.
This paper improves sample efficiency for off-policy evaluation with preference data.
problem Improving sample efficiency for off-policy evaluation with preference data.
method Using a deep neural network to learn the value function and leveraging manifold structure.
result Established a provably efficient guarantee for off-policy evaluation with RLHF.
Privacy-preserving reinforcement learning from human feedback using decoupled reward modeling.
problem Training large language models with sensitive user information while preserving privacy.
method Proposes a privacy-preserving framework that imposes differential privacy on reward learning only.
result Privacy contributes an additional additive term to the suboptimality gap, and the upper bound is rate-optimal up to logarithmic factors.
Efficiently identifies good policies by choosing contexts for human feedback.
problem Efficiently acquiring human feedback for preference alignment in large language models.
method Formalizes active exploration as a dueling bandit problem and proposes an active exploration algorithm with a polynomial worst-case regret bound.
result Proposed method outperforms baselines with limited human preferences on various language models and datasets.
Personalized size and fit recommendations bear crucial significance for any fashion e-commerce platform. Predicting the correct fit drives customer satisfaction and benefits the business by reducing costs incurred due to size-related returns. Traditional collaborative filtering algorithms seek to model customer prefere…
Meta-algorithm for efficient reinforcement learning from human preferences.
problem Learning from human preference comparisons in Markov decision processes.
method Randomized exploration and experimental design for batch comparison queries.
result Meta-algorithm achieves both regret and last-iterate guarantees with minimal preference queries.
A new method reduces the computational burden of safety alignment for large language models.
problem Safety concerns in large language models and the need to align them with human preferences.
method Optimal dualization approach to reduce constrained alignment to an unconstrained problem.
result Our algorithms MoCAN and PeCAN significantly reduce computational burden and improve training stability.
New approach uses inverse reinforcement learning to improve language model training.
problem Training large language models using imitation learning methods.
method Developed a new method of inverse reinforcement learning to optimize sequences directly.
result IRL-based fine-tuning leads to better performance and diversity in language generation.
Neural algorithms optimize arm selection with human preference feedback for complex reward functions.
problem Optimizing arm selection with noisy human preference feedback for complex, non-linear reward functions.
method Neural network to estimate reward function using preference feedback, upper confidence bound and Thompson sampling algorithms.
result Sub-linear regret guarantees for efficient arm selection in contextual dueling bandits.
The goal of task transfer in reinforcement learning is migrating the action policy of an agent to the target task from the source task. Given their successes on robotic action planning, current methods mostly rely on two requirements: exactly-relevant expert demonstrations or the explicitly-coded cost function on targe…