New model captures intransitive preferences without concave likelihood.
problem Complex human choices not accounted for by traditional models.
method Inspired by Condorcet method, Majority Vote model using RUMs.
result Three-dimensional model can represent strong, long intransitive cycles.
Modeling preference rankings with salient features to explain irrational choices.
problem Estimating rankings from noisy pairwise comparisons with irrational choices.
method Salient feature preference model with maximum likelihood estimation.
result Strong performance of maximum likelihood estimation on synthetic and real data.
We show that formal isomorphism of intransitive linear Lie equations along transversal to the orbits can be extended to neighborhoods of these transversal. In analytic cases, the word formal is dropped from theorems. Also, we associate an intransitive Lie algebra with each intransitive linear Lie equation, and from the…
Local classification of 4D Ricci solitons with specific algebra properties.
problem Classifying 4D Ricci solitons with a 2D Abelian Killing algebra.
method Local classification under specific curvature and symmetry conditions.
result Classification of Ricci solitons with orthogonally intransitive 2D Abelian Killing algebra.
A lemma of Tits establishes a connection between the simple connectivity of an incidence geometry and the universal completion of an amalgam induced by a sufficiently transitive group of automorphisms of that geometry. In the present paper, we generalize this lemma to intransitive geometries, thus opening the door for …
SLHF uses sequential game theory to optimize preferences from human feedback.
problem Optimizing preferences from human feedback in sequential settings.
method SLHF frames the problem as a sequential-move game between Leader and Follower, decomposing the optimization into refinement and adversarial optimization.
result SLHF achieves strong alignment across diverse preference datasets and scales to large models.
New method proves fast regret bounds for online RLHF with generalized preferences.
problem Minimizing max-regret in online RLHF with general preferences and bandit feedback.
method Adopted Generalized Bilinear Preference Model (GBPM) to investigate polylogarithmic regret guarantees.
result Proved polylogarithmic regret bounds for Greedy Sampling and Explore-Then-Commit policies under GBPM.
Investigates CR structures in 7D, showing 8 is max symmetry dimension.
problem CR structures in 7D with intransitive symmetry.
method Various methods to bound symmetry dimension, demonstrating existence of non-equivalent models.
result Existence of infinitely many non-equivalent submaximally symmetric models.
SPPO optimizes language model alignment by treating preferences as a game and achieving state-of-the-art performance.
problem Capturing intransitivity and irrationality in human preferences for accurate language model alignment.
method Self-play-based approach to identify Nash equilibrium policy through iterative policy updates.
result SPPO achieves state-of-the-art win-rate of 28.53% on AlpacaEval 2.0 without external supervision.
A new method for RLHF using proximal point Nash learning.
problem Capturing real human preferences in RLHF.
method Proximal point Nash learning, embedding self-play updates into a proximal point framework.
result High-probability last-iterate convergence for the combined method.
Elie Cartan's general equivalence problem is recast in the language of Lie algebroids. The resulting formalism, being coordinate and model-free, allows for a full geometric interpretation of Cartan's method of equivalence via reduction and prolongation. We show how to construct certain normal forms (Cartan algebroids) …
For surfaces, we brush a reasonably sharp picture of the influence of the fundamental group upon the complexity of foliated-dynamics. A metaphor emerges with phase-changes through the solid-liquid-gaseous states. Groups of ranks 0≤r≤1 are frozen with intransitivity reigning ubiquitously. When 2≤r≤3, th…
This paper is a continuation of Part I where the general setup was developed. Here we discuss the general equivalence problem for geometric structures and provide criteria for the equivalence, local and global, of transitive structures. Cartan's Flag Systems illustrate the theory as a major example and, finally, some a…
Flexible ranking models from choice data.
problem Difficulties in modeling, learning from, and predicting rankings.
method Choice-based ranking models using repeated selection.
result Choice-based ranking models outperform existing models in various ranking tasks.
Lie algebroids are by no means natural as an infinitesimal counterpart of groupoids. In this paper we propose a functorial construction called Nishimura algebroids for an infinitesimal counterpart of groupoids. Nishimura algebroids, intended for differential geometry, are of the same vein as Lawvere's functorial notion…
A general model for geometric structures on differentiable manifolds is obtained by deforming infinitesimal symmetries. Specifically, this model consists of a Lie algebroid, equipped with an affine connection compatible with the Lie algebroid structure. The curvature of this connection vanishes precisely when the struc…
In this paper we describe market in projective geometry language and give definition of a matrix of market rate, which is related to the matrix rate of return and the matrix of judgements in the Analytic Hierarchy Process (AHP). We use these observations to extend the AHP model to projective geometry formalism and gene…
A multiplicatively closed, horizontal n-plane field D on a Lie groupoid G over M generalizes to intransitive geometry the classical notion of a Cartan connection. The infinitesimalization of the connection D is a Cartan connection ∇ on the Lie algebroid of G, a notion already studied elsewhere by th…
Optimizes molecular generation for chemist preferences.
problem Models lack inherent preferences for chemist-desired structures.
method Fine-tuning with Direct Preference Optimization.
result Approach is simple, efficient, and highly effective.
New method adapts to user preferences dynamically, improving recommendation models.
problem Current recommendation models lack dynamic adaptation to changing user preferences.
method Preference Discerning with LLM-Enhanced Generative Retrieval
result Mender achieves state-of-the-art performance in adapting to evolving user preferences.
Investment and consumption strategies optimized with uncertain parameters.
problem Investment and consumption preferences in an incomplete financial market with uncertain parameters.
method PDE characterization and semi-explicit saddle-point construction of forward preferences and optimal strategies.
result A specific relationship between initial investment preference and forward consumption preference is necessary.
DPS uses posterior sampling for preference-based RL, achieving a first regret guarantee.
problem Formal frameworks for preference-based RL with theoretical analysis.
method Preference-based posterior sampling, Bayesian credit assignment.
result First asymptotic Bayesian no-regret rate for preference-based RL.
Many real-world engineering problems rely on human preferences to guide their design and optimization. We present PrefOpt, an open source package to simplify sequential optimization tasks that incorporate human preference feedback. Our approach extends an existing latent variable model for binary preferences to allow f…
Enhances preference learning by incorporating response times into binary choices.
problem Limited information from binary choices about preference strength.
method Combines choices and response times using the EZ diffusion model.
result Response times improve utility estimation for strong preferences.
Bayesian optimization learns DM preferences for multi-outcome experiments.
problem Optimizing expensive experiments with unknown utility functions and multiple outcomes.
method Alternates preference learning and Bayesian optimization, using pairwise comparisons.
result Preference exploration strategies improve Bayesian optimization performance.
New study shows personalized content recommendations can lead to polarization of user preferences.
problem Personalized content recommendations can alter user preferences, leading to polarization.
method Used a model of preference dynamics to explore how personalized content affects user preferences.
result Standard reward maximization algorithms achieve only constant regret in personalized recommendation environments.
Bayesian optimization agent learns user preferences from pairwise comparisons.
problem Learning user preferences from unknown and infinite choices.
method Sequential Bayesian optimization with pairwise comparisons.
result Optimal agent strategy minimizes remaining system uncertainty.
New RLHF framework handles general preference oracles without reward functions.
problem Handling general preference oracles without assuming a reward function.
method Developed a minimax game between two LLMs for RLHF under a general preference oracle, focusing on KL-regularized preference.
result Proposed algorithms for efficient offline and online RLHF learning.
Robot learns user preferences from brain signals.
problem Decoding user preferences for robot motions from brain signals.
method Proposes a novel approach using electroencephalography to decode user preferences from brain signals.
result Brain signals can reliably infer user preferences for robot trajectories.
Study on identifying most preferred policy in bandits with vector-valued rewards.
problem Identifying the most preferred policy in bandits with vector-valued rewards.
method Derive a novel lower bound on sample complexity, design the Preference-based Track and Stop (PreTS) algorithm, and derive a new concentration inequality.
result The sample complexity of PreTS is asymptotically tight.
Stable and consistent model alignment for language models without assuming human preference models.
problem Lack of statistical consistency in existing alignment methods.
method Relative density ratio optimization between preferred and mixture of preferred and non-preferred data distributions.
result Our approach achieves statistical consistency and stability, providing tighter convergence guarantees.
Dropping a tiny fraction of preferences can significantly alter the rankings of top LLMs.
problem Robustness of LLM ranking systems to small changes in preference data.
method A computational method based on the Bradley-Terry model to evaluate robustness.
result Top LLM rankings can be highly sensitive to the removal of a small fraction of preferences.
Paper explores limits and possibilities of aligning LLMs with human preferences.
problem Aligning LLMs with diverse human preferences to ensure fairness and informed outcomes.
method Analysis of probabilistic representation of human preferences and preservation of diverse preferences.
result LLMs can't fully align with human preferences using reward-based approaches due to Condorcet cycles, but mixed strategies are statistically possible.
Paper improves parameter estimation of continuous distributions using preference feedback.
problem Improving parameter estimation of continuous distributions.
method Preference-based M-estimators and deterministic preferences.
result Preference-based estimators achieve an estimation error scaling of O(1/n), significantly faster than sample-only methods.
This work aims to separate buying preferences from merchandise commercials in fashion e-retail.
problem Difficult to infer customer preference from sales data due to implicit signals.
method Extends earlier work on explicit signals to implicit signals from user behavior.
result Derives a metric to separate buying preferences from merchandise commercials.
Paper investigates monotonicity issues in AI preference learning.
problem AI models may violate monotonicity when learning preferences.
method Investigates root causes of non-monotonicity in comparison-based preference learning.
result Proves local pairwise monotonicity under mild assumptions.
DOPL learns from preference feedback to solve RMAB problems.
problem Learning optimal decisions in RMAB with limited reward information.
method Direct online preference learning (DOPL) for Pref-RMAB.
result DOPL achieves sublinear regret for RMAB with preference feedback.
Direct Density Ratio Optimization aligns LLMs with human preferences without assuming specific models.
problem Statistical inconsistency in aligning LLMs with human preferences.
method Direct Density Ratio Optimization (DDRO) estimates density ratio directly.
result DDRO is statistically consistent, converging to true human preferences as data grows.
Bayesian optimization with preference learning identifies preferred solutions in multi-objective problems.
problem Optimizing multiple criteria with decision maker preferences in expensive functions.
method Bayesian optimization with interactive preference learning and active acquisition function.
result Identifies the most preferred solution with reduced interaction cost.
CPMetric learns distances between structured preferences using deep neural networks.
problem Learning distances between structured preference representations.
method Deep Siamese Networks and CP-net formalism for metric learning.
result CPDist outperforms existing approximation algorithms in accuracy and computation time.
Training models to prefer certain responses can unintentionally shift probability to harmful ones.
problem Likelihood displacement in DPO models, leading to unintended unalignment.
method Characterized and mitigated likelihood displacement using CHES score.
result Training models to prefer certain responses can unintentionally shift probability mass to harmful responses.
This work proves win rate is key to understanding preference learning.
problem Understanding preference learning from generative models.
method Analyzing preference learning methods as win rate optimization or non-WRO.
result Proves win rate is the only evaluation respecting preferences and prevalences.
IDT learns human preferences from uncertain decisions, even when humans are suboptimal.
problem Learning human preferences from uncertain and suboptimal decisions.
method Inverse decision theory (IDT) framework, statistical analysis of IDT, characterizing sample complexity.
result Learning preferences is easier when decisions are more uncertain, even if humans are suboptimal.
The paper proposes a method to infer multi-objective rewards from preferences.
problem Modeling preferences based on multiple, often competing objectives.
method Modeling priorities lexicographically and inferring multi-objective rewards from observed preferences.
result Lexicographically-ordered rewards provide a better understanding of preferences and improve policies.
The paper explores game-theoretic alignment of LLMs with human preferences, finding limitations and conditions.
problem Aligning LLMs with human preferences using game theory.
method Systematic study of payoff choices in a two-player zero-sum game for desirable alignment properties.
result Impossibility of preference matching in game-theoretic LLM alignment under standard assumptions.
Bal-PM reduces preference labeling costs for LLMs.
problem Efficiently acquiring human feedback for preference modeling in large language models.
method Bayesian Active Learning with entropy maximization in feature space.
result Bal-PM reduces the number of required preference labels by 33% to 68%.
DPA aligns LLMs with multi-objective rewards for diverse user preferences.
problem Fine-grained control over LLMs for diverse user needs.
method Integrates multi-objective reward modeling and directional preference control.
result DPA offers better performance trade-offs and intuitive user control over LLM generation.
Tutorials on preference learning with Gaussian Processes.
problem Understanding individual preferences and choices for efficient and personalized applications.
method Presentation of a comprehensive framework for preference learning with Gaussian Processes, incorporating rationality principles.
result Construction of preference learning models that encompass various utility models and scenarios.