The paper explores game-theoretic alignment of LLMs with human preferences, finding limitations and conditions.
problem Aligning LLMs with human preferences using game theory.
method Systematic study of payoff choices in a two-player zero-sum game for desirable alignment properties.
result Impossibility of preference matching in game-theoretic LLM alignment under standard assumptions.
Direct Density Ratio Optimization aligns LLMs with human preferences without assuming specific models.
problem Statistical inconsistency in aligning LLMs with human preferences.
method Direct Density Ratio Optimization (DDRO) estimates density ratio directly.
result DDRO is statistically consistent, converging to true human preferences as data grows.
Paper explores limits and possibilities of aligning LLMs with human preferences.
problem Aligning LLMs with diverse human preferences to ensure fairness and informed outcomes.
method Analysis of probabilistic representation of human preferences and preservation of diverse preferences.
result LLMs can't fully align with human preferences using reward-based approaches due to Condorcet cycles, but mixed strategies are statistically possible.
Extends reinforcement learning alignment to scalar rewards, improving math reasoning.
problem Designing reinforcement learning algorithms for general LLM alignment.
method Introduces f-GRPO and f-HAL, estimating f-divergences between reward-aligned and unaligned distributions.
result Improves math reasoning RLVR tasks and mitigates reward hacking.
PIPA aligns preferences without reinforcement learning, improving language model performance.
problem Aligning preferences in language models efficiently and without reinforcement learning.
method Formulates preference alignment as a Maximum Likelihood Estimation problem with prior constraints.
result PIPA algorithms achieve up to 10% performance improvement on benchmarks.
The paper addresses poor calibration in fine-tuned LLMs after preference alignment.
problem Poor calibration in fine-tuned Large Language Models (LLMs) after preference alignment.
method Proposes a calibration-aware fine-tuning approach to restore calibration without compromising model performance.
result Demonstrates the effectiveness of the proposed methods through extensive experiments.
Stable and consistent model alignment for language models without assuming human preference models.
problem Lack of statistical consistency in existing alignment methods.
method Relative density ratio optimization between preferred and mixture of preferred and non-preferred data distributions.
result Our approach achieves statistical consistency and stability, providing tighter convergence guarantees.
AOT aligns LLMs on distributional preferences via optimal transport.
problem Current LLM alignment techniques lack distributional level alignment.
method Alignment via Optimal Transport (AOT) aligns LLMs on unpaired preference data.
result AOT enables alignment by penalizing reward distribution violations.
New RLHF approach mitigates bias in aligning LLMs with human preferences.
problem Algorithmic bias in RLHF leading to preference collapse.
method Preference Matching (PM) RLHF, using PM regularizer and conditional variant.
result 29% to 41% improvement in alignment with human preferences.
Efficiently identifies good policies by choosing contexts for human feedback.
problem Efficiently acquiring human feedback for preference alignment in large language models.
method Formalizes active exploration as a dueling bandit problem and proposes an active exploration algorithm with a polynomial worst-case regret bound.
result Proposed method outperforms baselines with limited human preferences on various language models and datasets.
DPA aligns LLMs with multi-objective rewards for diverse user preferences.
problem Fine-grained control over LLMs for diverse user needs.
method Integrates multi-objective reward modeling and directional preference control.
result DPA offers better performance trade-offs and intuitive user control over LLM generation.
A new method reduces preference distortion in LLM alignment.
problem Vulnerability of traditional LLM alignment methods to human preference heterogeneity.
method Sign Estimator: A simple, provably consistent, and efficient estimator using binary classification loss.
result Substantially reduces preference distortion over a panel of simulated personas.
Proposes a robust algorithm for aligning large language models with human preferences.
problem Misspecification in preference models, reference policies, and reward functions.
method Doubly robust preference optimization algorithm.
result Superior and more robust performance compared to state-of-the-art algorithms.
Activists align with large fund preferences for success.
problem Aligning with large fund preferences increases activist success.
method Analyzed previous proxy voting behavior to estimate preferences and correlated them with activist success.
result Campaigns with higher alignment receive more votes and are more successful.
LLM safety alignment explained as divergence estimation.
problem Aligning large language models to avoid harmful outputs.
method Presented a theoretical framework showing alignment methods as divergence estimators.
result KLDO method improves safety alignment using compliance-refusal datasets.
Paper investigates monotonicity issues in AI preference learning.
problem AI models may violate monotonicity when learning preferences.
method Investigates root causes of non-monotonicity in comparison-based preference learning.
result Proves local pairwise monotonicity under mild assumptions.
PILAF optimizes reward models from human feedback for better policy alignment.
problem Creating accurate reward models from human feedback for policy optimization.
method Policy-Interpolated Learning for Aligned Feedback (PILAF) that explicitly aligns preference learning with maximizing underlying oracle reward.
result PILAF is optimal from both optimization and statistical perspectives, demonstrating strong performance in RLHF settings.
SARA uses similarity to learn rewards robustly and adaptively.
problem Robustness to labeler errors and adaptability to diverse feedback formats.
method Contrastive framework that learns latent representations and computes rewards as similarities.
result Strong performance on offline RL benchmarks and diverse applications.
Unified framework for aligning LLMs from human feedback.
problem Lack of strong theoretical justification for RLHF and inability to compare methods.
method Reframed alignment as distribution learning from pairwise preferences, proposing three principled objectives.
result Proposed objectives achieve strong non-asymptotic convergence to target LM.
Response time improves alignment with diverse human preferences.
problem Standard aggregation of feedback ignores heterogeneity and anonymity.
method Augmenting feedback with response time data and modeling decisions with DDM.
result Estimator of heterogeneous preferences converges to true average preference.
Develops methods to correct bias in AI feedback for more accurate alignment.
problem Systematic bias in AI feedback compared to human labels.
method Two debiased alignment methods: DDPO and DIPO.
result Methods improve alignment efficiency and performance close to human-labeled data.
Paper shows equivalence between two alignment methods and introduces a new algorithm.
problem Ensuring human alignment of large language models for useful, safe, and pleasant user experience.
method Introduces IPO-MD algorithm, showing equivalence between IPO and Nash-MD methods.
result Equivalence between IPO and Nash-MD methods proven when considering online version of IPO.
Paper addresses reward hacking in preference optimization, proposing POWER-DL to improve AI alignment.
problem Reward hacking problem in preference optimization, leading to undesired behaviors.
method POWER-DL combines robust reward maximization and dynamic label updates to mitigate reward hacking.
result POWER-DL consistently outperforms state-of-the-art methods on alignment benchmarks.
RCPO uses ranked choice modeling for better LLM alignment.
problem Pairwise preference optimization limits LLM alignment.
method Unified framework combining preference optimization and ranked choice modeling.
result RCPO outperforms competitive baselines in LLM alignment.
SPPO optimizes language model alignment by treating preferences as a game and achieving state-of-the-art performance.
problem Capturing intransitivity and irrationality in human preferences for accurate language model alignment.
method Self-play-based approach to identify Nash equilibrium policy through iterative policy updates.
result SPPO achieves state-of-the-art win-rate of 28.53% on AlpacaEval 2.0 without external supervision.
Novel PO algorithms improve LLM alignment tasks.
problem Evaluating PO algorithms on LLM alignment is costly and noisy.
method Designed a diagnostic suite of MuJoCo tasks and datasets, used evolutionary strategies to discover specialized PO algorithms.
result Proposed MPO algorithms significantly outperform existing PO algorithms in LLM alignment.
SAIL improves online alignment of large language models with minimal feedback.
problem Offline RLHF methods often lead to sub-optimal performance due to fixed preference datasets.
method SAIL uses bilevel optimization and a single-level first-order method to iteratively refine model alignment.
result SAIL significantly improves alignment performance on open-sourced datasets with minimal computational overhead.
Paper addresses online alignment of large language models under uncertain preference feedback.
problem Online alignment of large language models with misspecified preference feedback.
method Formulates an oracle-robust objective as a worst-case optimization problem for log-linear policies, and develops projected stochastic composite updates.
result Shows that the robust objective admits an exact closed-form decomposition and achieves O ~ ( ε − 2 ) \widetilde{O}(\varepsilon^{-2}) O ( ε − 2 ) oracle complexity. This work frames reward modelling from preferences as a causal problem.
problem Reward modelling from preference data for AI alignment.
method Causal inference approach to identify challenges and assumptions.
result Causally-inspired approaches improve model robustness.
RLHF performs well despite violating social choice theory axioms.
problem RLHF's empirical success contradicts social choice theory axioms.
method Showed RLHF satisfies pairwise majority and Condorcet consistency under mild assumptions, and introduced new alignment criteria.
result RLHF satisfies pairwise majority and Condorcet consistency under mild assumptions, explaining its practical success.
Paper connects Plackett-Luce and Cox models for preference estimation.
problem Estimating preferences from annotated data.
method Connects Plackett-Luce model to Cox Proportional Hazards model.
result Implications of the connection between the two models.
EBRM improves robustness and generalization of language model rewards.
problem Challenges in capturing complex human preferences and generalizing to unseen data in reward models.
method Energy-Based Reward Model (EBRM) that models reward distribution explicitly, using conflict-aware data filtering, label-noise-aware contrastive training, and hybrid initialization.
result Significant improvements in robustness and generalization, up to 5.97% improvement in safety-critical alignment tasks.
This paper introduces f f f -DPO, a generalized approach to Direct Preference Optimization using diverse divergence constraints.
problem Aligning large language models with human preferences while mitigating safety risks.
method Incorporates diverse divergence constraints to simplify the relationship between reward and optimal policy, eliminating the need for estimating the normalizing constant.
result Optimizes LLMs to align with human preferences more efficiently and under a broader set of divergence constraints.
A Markov Chain approach for aligning generative models from pairwise human preferences.
problem Aligning generative models from pairwise human preferences.
method Markov Chain from Human Feedback (MCHF)
result MCHF converges geometrically fast to the stationary distribution.
Optimizes molecular generation for chemist preferences.
problem Models lack inherent preferences for chemist-desired structures.
method Fine-tuning with Direct Preference Optimization.
result Approach is simple, efficient, and highly effective.
Demon aligns diffusion models without retraining or backpropagation.
problem Aligning diffusion models with user preferences.
method Stochastic optimization to control noise distribution.
result Significantly improves aesthetics scores for text-to-image generation.
DRO-REBEL improves LLM alignment by robustly updating models online.
problem Overfitting and drifting of LLMs during RLHF.
method DRO-REBEL uses type- p p p Wasserstein, KL, and χ 2 χ^2 χ 2 ambiguity sets for robust online updates. result DRO-REBEL achieves faster convergence and better performance than prior methods.
Mitigates overoptimization in RLHF by reformulating SFT loss as a preference optimization loss.
problem Overoptimization in RLHF where reward model misguides generative model.
method Proposes a theoretical algorithm that minimizes maximum likelihood estimation and reward penalty, reformulates as simple objective combining preference optimization and supervised learning losses.
result Improved performance of RPO compared to DPO baselines in aligning LLMs.
New method optimizes policies without assuming known link functions between preferences and rewards.
problem Policy alignment with unknown and unrestricted link functions.
method Formulates an f f f -divergence-constrained reward maximization problem, learning policies directly. result Induces a semiparametric single-index binary choice model for policy alignment.
Hölder-DPO aligns models robustly with noisy human feedback.
problem No existing alignment methods can handle severe label noise.
method Proposes Hölder-DPO, a principled alignment loss with provable redescending property.
result Hölder-DPO enables scalable human feedback valuation and improves model alignment.
Survey of alignment techniques for large language models.
problem Ensuring large language models align with human values.
method Analysis of diverse alignment methods and training paradigms.
result Preference-based methods offer more flexibility for nuanced alignment.
This work proves win rate is key to understanding preference learning.
problem Understanding preference learning from generative models.
method Analyzing preference learning methods as win rate optimization or non-WRO.
result Proves win rate is the only evaluation respecting preferences and prevalences.
EAST aligns neural network classifiers with user-defined evaluation metrics.
problem Mismatch between neural network training and evaluation metrics leads to suboptimal performance.
method EAST uses dynamic thresholding, soft-set confusion matrix, and annealing to align neural network predictions with target evaluation metrics.
result EAST improves alignment between training objectives and evaluation metrics, outperforming existing methods.
A new method reduces the computational burden of safety alignment for large language models.
problem Safety concerns in large language models and the need to align them with human preferences.
method Optimal dualization approach to reduce constrained alignment to an unconstrained problem.
result Our algorithms MoCAN and PeCAN significantly reduce computational burden and improve training stability.
QAlign improves language model alignment with less compute, outperforming existing methods.
problem Improving language model performance with limited test-time computation.
method QAlign: sampling from optimal aligned distribution using Markov chain Monte Carlo.
result Consistent improvements over existing methods on various benchmarks.
Training models to prefer certain responses can unintentionally shift probability to harmful ones.
problem Likelihood displacement in DPO models, leading to unintended unalignment.
method Characterized and mitigated likelihood displacement using CHES score.
result Training models to prefer certain responses can unintentionally shift probability mass to harmful responses.
SLHF uses sequential game theory to optimize preferences from human feedback.
problem Optimizing preferences from human feedback in sequential settings.
method SLHF frames the problem as a sequential-move game between Leader and Follower, decomposing the optimization into refinement and adversarial optimization.
result SLHF achieves strong alignment across diverse preference datasets and scales to large models.
Paper proposes f-DPG for aligning language models with preferences.
problem Aligning language models with user preferences.
method Uses f-divergence to approximate target distributions and minimizes a forward KL from it using DPG.
result Jensen-Shannon divergence often outperforms forward KL divergence, leading to significant improvements.