Direct Density Ratio Optimization aligns LLMs with human preferences without assuming specific models.
problem Statistical inconsistency in aligning LLMs with human preferences.
method Direct Density Ratio Optimization (DDRO) estimates density ratio directly.
result DDRO is statistically consistent, converging to true human preferences as data grows.
Optimizes molecular generation for chemist preferences.
problem Models lack inherent preferences for chemist-desired structures.
method Fine-tuning with Direct Preference Optimization.
result Approach is simple, efficient, and highly effective.
Improved DPO framework penalizes preference uncertainty to avoid overoptimization.
problem Aligning LLMs to human preferences is challenging due to varied, context-dependent, and ambiguous preferences.
method Developed a pessimistic framework for DPO by introducing preference uncertainty penalization schemes.
result Improved overall performance and better completions on high-uncertainty responses compared to vanilla DPO.
New methods improve LLM preference optimization by intelligently weighting multiple reference models.
problem Improving LLM preference optimization with multiple reference models.
method Introducing four new weighting strategies for multiple-reference preference optimization.
result All four new weighting strategies outperform current methods on preference accuracy.
Active learning framework for optimizing human preferences in reinforcement learning.
problem Selecting most informative feedback for training models of human preferences.
method Proposes an active learning framework to collect preferential feedback online or offline.
result Errors in DPO logit estimates diminish with more feedback.
This paper introduces f-DPO, a generalized approach to Direct Preference Optimization using diverse divergence constraints.
problem Aligning large language models with human preferences while mitigating safety risks.
method Incorporates diverse divergence constraints to simplify the relationship between reward and optimal policy, eliminating the need for estimating the normalizing constant.
result Optimizes LLMs to align with human preferences more efficiently and under a broader set of divergence constraints.
Training models to prefer certain responses can unintentionally shift probability to harmful ones.
problem Likelihood displacement in DPO models, leading to unintended unalignment.
method Characterized and mitigated likelihood displacement using CHES score.
result Training models to prefer certain responses can unintentionally shift probability mass to harmful responses.
DPA aligns LLMs with multi-objective rewards for diverse user preferences.
problem Fine-grained control over LLMs for diverse user needs.
method Integrates multi-objective reward modeling and directional preference control.
result DPA offers better performance trade-offs and intuitive user control over LLM generation.
A-GPS learns to generate Pareto sets efficiently with user preferences.
problem Online discrete multi-objective optimization with user preferences.
method Generative model with class probability estimator (CPE) for non-dominance and preference alignment.
result Amortized generative model for efficient Pareto set approximation.
Optimizes explanations for better listener understanding.
problem Insufficient consideration of listener preferences in concept-based explanations.
method Iterative training procedure based on direct preference optimization.
result Pragmatic explanations improve both model accuracy and user understanding.
Enhances math problem-solving models with multi-turn preference learning.
problem Improving mathematical problem-solving capabilities of large language models.
method Introduces a multi-turn direct preference learning framework for tool-integrated mathematical reasoning tasks.
result Significant performance improvements in model accuracy on math datasets.
Paper investigates monotonicity issues in AI preference learning.
problem AI models may violate monotonicity when learning preferences.
method Investigates root causes of non-monotonicity in comparison-based preference learning.
result Proves local pairwise monotonicity under mild assumptions.
Stable and consistent model alignment for language models without assuming human preference models.
problem Lack of statistical consistency in existing alignment methods.
method Relative density ratio optimization between preferred and mixture of preferred and non-preferred data distributions.
result Our approach achieves statistical consistency and stability, providing tighter convergence guarantees.
Paper connects Plackett-Luce and Cox models for preference estimation.
problem Estimating preferences from annotated data.
method Connects Plackett-Luce model to Cox Proportional Hazards model.
result Implications of the connection between the two models.
MallowsPO enhances LLM fine-tuning with a dispersion index of human preferences.
problem Lack of diversity in human preferences in DPO.
method Developed a dispersion index based on Mallows' theory to characterize preference diversity.
result Demonstrated improved performance in various tasks using the dispersion index.
Derives a new objective to learn from human preferences without approximations.
problem Learning from human preferences through RLHF relies on approximations that can lead to pitfalls.
method Derives a new general objective ΨPO that bypasses both approximations. result Demonstrates the superiority of the new objective to Direct Preference Optimisation (DPO) empirically.
Paper develops methods to optimize policies directly from human feedback without reward inference.
problem Challenges in RLHF, including reward model overfitting and distribution shift.
method Develops two algorithms for RLHF without reward inference, using zeroth-order gradient approximators.
result Establishes polynomial convergence rates and outperforms existing methods in numerical experiments.
Unified approach to RLHF tackles uncertainty in reward function.
problem Uncertainty in reward function learned from human feedback.
method Value-incentivized preference optimization (VPO) that regularizes the reward function with value function.
result Theoretical and practical guarantees for both online and offline RLHF settings.
MAXMINLCB optimizes unknown target functions with preference feedback using a Stackelberg game approach.
problem Optimizing unknown target functions with pairwise comparisons and human feedback.
method MAXMINLCB, a zero-sum Stackelberg game, balances exploration and exploitation.
result MAXMINLCB consistently outperforms existing algorithms with a rate-optimal regret guarantee.
Bayesian optimization (BO) has emerged during the last few years as an effective approach to optimizing black-box functions where direct queries of the objective are expensive. In this paper we consider the case where direct access to the function is not possible, but information about user preferences is. Such scenari…
FinDPO uses preference optimization to improve financial sentiment analysis models.
problem Financial sentiment analysis models often fail to generalize to unseen data.
method FinDPO uses Direct Preference Optimization (DPO) to align LLMs with human preferences.
result FinDPO achieves state-of-the-art performance and maintains positive returns under realistic trading conditions.
Study optimal consumption and investment for investors with Epstein-Zin preferences.
problem Optimal consumption and investment for investors with Epstein-Zin preferences in an incomplete market.
method Variational characterisation and direct method to prove existence of optimal policies.
result Existence and uniqueness of optimal consumption and investment policies.
PBO framework optimizes latent preferences over multiple objectives.
problem Optimizing latent preferences with multiple conflicting objectives.
method Proposes DSTS, a multi-objective generalization of dueling Thompson sampling.
result DSTS outperforms benchmarks and provides asymptotic consistency.
Mitigates overoptimization in RLHF by reformulating SFT loss as a preference optimization loss.
problem Overoptimization in RLHF where reward model misguides generative model.
method Proposes a theoretical algorithm that minimizes maximum likelihood estimation and reward penalty, reformulates as simple objective combining preference optimization and supervised learning losses.
result Improved performance of RPO compared to DPO baselines in aligning LLMs.
EXPO framework eliminates need for reward model, achieving better optimization.
problem Optimizing large language model responses without a separate reward model.
method Introduces EXPO framework that avoids reparameterization, directly optimizing preferences.
result Demonstrates better regularization and intuitive interpolation behaviors.
PIPA aligns preferences without reinforcement learning, improving language model performance.
problem Aligning preferences in language models efficiently and without reinforcement learning.
method Formulates preference alignment as a Maximum Likelihood Estimation problem with prior constraints.
result PIPA algorithms achieve up to 10% performance improvement on benchmarks.
A new algorithm RTO integrates DPO and PPO for RLHF, improving performance.
problem Learning from sparse, sentence-level rewards in RLHF.
method Modeling RLHF as MDP, introducing RTO that learns token-wise reward function.
result RTO outperforms PPO on benchmarks by 7.5 points on AlpacaEval 2 and 4.1 points on Arena-Hard.
ZSPO optimizes RL from unknown link functions using human feedback.
problem Designing RLHF algorithms for unknown link functions.
method Zero-order policy optimization with human preference feedback.
result ZSPO converges to a stationary policy with a polynomial rate.
ADPO optimizes relative advantage in reinforcement learning from human feedback.
problem Optimizing policy alignment in reinforcement learning from human preferences.
method ADPO explicitly parameterizes the optimal structure through anchored logits, decoupling response quality from prior popularity.
result Empirically, ADPO achieves state-of-the-art performance on reasoning tasks, outperforming GRPO by 30.9 percent.
Paper addresses RLHF alignment challenges with novel algorithms.
problem Challenges in RLHF alignment, especially in strategic exploration.
method Develops a reverse-KL regularized contextual bandit formulation and proposes efficient algorithms with theoretical guarantees.
result Proposed methods significantly outperform existing RLHF algorithms in real-world experiments.
Variational Optimization forms a differentiable upper bound on an objective. We show that approaches such as Natural Evolution Strategies and Gaussian Perturbation, are special cases of Variational Optimization in which the expectations are approximated by Gaussian sampling. These approaches are of particular interest …
In the presence of ambiguity on the driving force of market randomness, we consider the dynamic portfolio choice without any predetermined investment horizon. The investment criteria is formulated as a robust forward performance process, reflecting an investor's dynamic preference. We show that the market risk premium …
Efficiently identifies good policies by choosing contexts for human feedback.
problem Efficiently acquiring human feedback for preference alignment in large language models.
method Formalizes active exploration as a dueling bandit problem and proposes an active exploration algorithm with a polynomial worst-case regret bound.
result Proposed method outperforms baselines with limited human preferences on various language models and datasets.
Efficiently poisons offline RLHF models by flipping preference labels.
problem Vulnerability of offline RLHF models to preference label flipping attacks.
method Developed two attack methods: BAL-A and BMP-A, solving a structured binary sparse approximation problem.
result Demonstrated that flipping one preference label induces a parameter-independent shift in the DPO gradient, enabling structured binary sparse approximation.
Develops methods to correct bias in AI feedback for more accurate alignment.
problem Systematic bias in AI feedback compared to human labels.
method Two debiased alignment methods: DDPO and DIPO.
result Methods improve alignment efficiency and performance close to human-labeled data.
We consider the problem of identifying the most profitable product design from a finite set of candidates under unknown consumer preference. A standard approach to this problem follows a two-step strategy: First, estimate the preference of the consumer population, represented as a point in part-worth space, using an ad…
A particular Finsler-metric proposed in [1,2] and describing a geometry with a preferred null direction is characterized here as belonging to a subclass contained in a larger class of Finsler-metrics with one or more preferred directions (null, space- or timelike). The metrics are classified according to their group of…
New method for efficient online exploration in RLHF reduces regret.
problem Efficiently collecting new preference data in RLHF to refine reward model and policy.
method Proposes a new exploration scheme that directs preference queries toward reducing uncertainty in reward differences most relevant to policy improvement.
result Establishes regret bounds of order T(β+1)/(β+2) for online RLHF, with polynomial scaling in all model parameters. DOPL learns from preference feedback to solve RMAB problems.
problem Learning optimal decisions in RMAB with limited reward information.
method Direct online preference learning (DOPL) for Pref-RMAB.
result DOPL achieves sublinear regret for RMAB with preference feedback.
Paper proposes efficient RLHF methods for LLMs using active queries.
problem Efficiently aligning LLMs with human preferences using RLHF.
method Formalizes RLHF as a dueling bandit problem, introduces APPO and ADPO algorithms.
result ADPO achieves similar performance to state-of-the-art methods with fewer queries.
This paper solves optimal consumption-investment problems with time-varying preferences.
problem Optimal consumption-investment problems under time-varying incomplete preferences.
method Develops a martingale-type solution in a topological vector space, using stochastic processes and scalarization methods.
result Optimal investment policies are set-valued, with selectors decomposed into four components.
New method improves consistency in preference learning for neural networks.
problem Inconsistent surrogate losses in preference learning for neural networks.
method Formulated a margin-shifted ranking framework and introduced Structure-Aware H-consistency. result Proved superior consistency guarantees for capacity-bounded models using heavy-tailed surrogates.
We study the active learning problem of top-k ranking from multi-wise comparisons under the popular multinomial logit model. Our goal is to identify the top-k items with high probability by adaptively querying sets for comparisons and observing the noisy output of the most preferred item from each comparison. To ac…
Algorithm tackles large-scale portfolio optimization with higher moments, improving computational efficiency.
problem Optimizing portfolios with higher moments (variance, skewness, kurtosis) for large asset universes is computationally infeasible.
method Developed a structure-exploiting algorithm based on Yau's affine-normal descent, working directly with return matrix.
result Algorithm avoids explicit higher-order tensors and exploits quartic structure for efficient computation.
By specifying model free preferences towards simple nested classes of lottery pairs, we develop the dual story to stand on equal footing with that of (primal) risk apportionment. The dual story provides an intuitive interpretation, and full characterization, of dual counterparts of such concepts as prudence and tempera…
This work simplifies data valuation for LLMs using Shapley value computation.
problem How to fairly distribute benefits from training superior LLMs with multiple data owners' resources.
method We leverage the specific mathematical structure of DPO to enable scalable Shapley value computation for LLMs.
result We demonstrate that Shapley value computation for LLMs trained with DPO is significantly simplified.
Duel-Evolve uses LLM self-preferences for test-time optimization of discrete outputs.
problem Optimizing LLM outputs at test time with limited or unreliable scalar rewards.
method Duel-Evolve uses pairwise comparisons from the LLM to guide optimization, aggregating them via a Bayesian Bradley-Terry model.
result Achieves significant improvement over existing methods in accuracy.
Bayesian optimization learns DM preferences for multi-outcome experiments.
problem Optimizing expensive experiments with unknown utility functions and multiple outcomes.
method Alternates preference learning and Bayesian optimization, using pairwise comparisons.
result Preference exploration strategies improve Bayesian optimization performance.