AOT aligns LLMs on distributional preferences via optimal transport.
problem Current LLM alignment techniques lack distributional level alignment.
method Alignment via Optimal Transport (AOT) aligns LLMs on unpaired preference data.
result AOT enables alignment by penalizing reward distribution violations.
Polestar optimizes public transportation routes for efficiency and user satisfaction.
problem Difficulty in finding optimal public transportation routes due to complex networks and dynamic situations.
method Developed a Public Transportation Graph (PTG) and a route search algorithm with station binding and ranking modules.
result Demonstrated superior efficiency and user satisfaction compared to existing systems.
New method creates synthetic pseudo-panel data to study transport preferences.
problem Aggregation bias in cross-sectional data.
method Conditional Variational Autoencoder (CVAE) framework for probabilistic model.
result Revealed dynamics of transport preferences and categorized individuals.
PMD measures user distances using optimal transportation, improving recommendation accuracy.
problem Sparse data and lack of co-rated items in collaborative filtering.
method PMD is an Earth Mover's Distance-based user distance measure that utilizes all ratings.
result PMD achieves superior recommendation accuracy, especially in sparse data scenarios.
Extends Optimal Transport to multiple agents, aiming for equitable and optimal distribution.
problem Sharing costs or goods equitably among multiple agents with different preferences.
method Minimizes the maximum transportation cost or maximizes the minimum utility.
result Provides a new algorithm faster than standard linear programming.
New method estimates sparse covariance matrices in logit mixtures.
problem Estimating correlations among random coefficients in logit models.
method Mixed-integer optimization (MIO) with Markov Chain Monte Carlo (MCMC) for posterior draws.
result Correctly recovers true covariance structure from synthetic data.
In this series of lectures we introduce the Monge-Kantorovich problem of optimally transporting one distribution of mass onto another, where optimality is measured against a cost function c(x,y). Connections to geometry, inequalities, and partial differential equations will be discussed, focusing in particular on recen…
New method assesses multivariate stochastic dominance using Optimal Transport.
problem Benchmarking models across multiple metrics considering dependencies.
method Characterization of multivariate first stochastic dominance via couplings, entropic regularization, and Optimal Transport.
result Established CLT and consistency for the empirical statistic, enabling hypothesis testing.
DeepGSB solves MFGs with non-differentiable preferences.
problem Solving MFGs with non-differentiable preferences and exact population convergence.
method Generalized Schrödinger Bridge via Forward-Backward SDEs and Temporal Difference learning.
result DeepGSB provides necessary and sufficient conditions for mean-field problems.
LLM evaluation suffers from systematic biases and lacks reliable positive judgments.
problem LLM evaluation suffers from systematic biases and lacks reliable positive judgments.
method Formulate LLM evaluation as a positive-unlabelled learning problem and propose a geometric auditing framework based on Partial Optimal Transport.
result Improved alignment with human preferences, increased robustness to presentation biases, and interpretable confidence estimates.
CTGAN synthesizes population data for travel behavior simulation.
problem Synthesizing population data for agent-based transportation modeling.
method Composite Travel Generative Adversarial Network (CTGAN).
result Consistent and accurate generation of synthetic populations with tabular and sequential mobility data.
When tracking user-specific online activities, each user's preference is revealed in the form of choices and comparisons. For example, a user's purchase history is a record of her choices, i.e. which item was chosen among a subset of offerings. A user's preferences can be observed either explicitly as in movie ratings …
Optimizes molecular generation for chemist preferences.
problem Models lack inherent preferences for chemist-desired structures.
method Fine-tuning with Direct Preference Optimization.
result Approach is simple, efficient, and highly effective.
New method adapts to user preferences dynamically, improving recommendation models.
problem Current recommendation models lack dynamic adaptation to changing user preferences.
method Preference Discerning with LLM-Enhanced Generative Retrieval
result Mender achieves state-of-the-art performance in adapting to evolving user preferences.
Investment and consumption strategies optimized with uncertain parameters.
problem Investment and consumption preferences in an incomplete financial market with uncertain parameters.
method PDE characterization and semi-explicit saddle-point construction of forward preferences and optimal strategies.
result A specific relationship between initial investment preference and forward consumption preference is necessary.
DPS uses posterior sampling for preference-based RL, achieving a first regret guarantee.
problem Formal frameworks for preference-based RL with theoretical analysis.
method Preference-based posterior sampling, Bayesian credit assignment.
result First asymptotic Bayesian no-regret rate for preference-based RL.
Many real-world engineering problems rely on human preferences to guide their design and optimization. We present PrefOpt, an open source package to simplify sequential optimization tasks that incorporate human preference feedback. Our approach extends an existing latent variable model for binary preferences to allow f…
Enhances preference learning by incorporating response times into binary choices.
problem Limited information from binary choices about preference strength.
method Combines choices and response times using the EZ diffusion model.
result Response times improve utility estimation for strong preferences.
Bayesian optimization learns DM preferences for multi-outcome experiments.
problem Optimizing expensive experiments with unknown utility functions and multiple outcomes.
method Alternates preference learning and Bayesian optimization, using pairwise comparisons.
result Preference exploration strategies improve Bayesian optimization performance.
New study shows personalized content recommendations can lead to polarization of user preferences.
problem Personalized content recommendations can alter user preferences, leading to polarization.
method Used a model of preference dynamics to explore how personalized content affects user preferences.
result Standard reward maximization algorithms achieve only constant regret in personalized recommendation environments.
Bayesian optimization agent learns user preferences from pairwise comparisons.
problem Learning user preferences from unknown and infinite choices.
method Sequential Bayesian optimization with pairwise comparisons.
result Optimal agent strategy minimizes remaining system uncertainty.
New RLHF framework handles general preference oracles without reward functions.
problem Handling general preference oracles without assuming a reward function.
method Developed a minimax game between two LLMs for RLHF under a general preference oracle, focusing on KL-regularized preference.
result Proposed algorithms for efficient offline and online RLHF learning.
Robot learns user preferences from brain signals.
problem Decoding user preferences for robot motions from brain signals.
method Proposes a novel approach using electroencephalography to decode user preferences from brain signals.
result Brain signals can reliably infer user preferences for robot trajectories.
Study on identifying most preferred policy in bandits with vector-valued rewards.
problem Identifying the most preferred policy in bandits with vector-valued rewards.
method Derive a novel lower bound on sample complexity, design the Preference-based Track and Stop (PreTS) algorithm, and derive a new concentration inequality.
result The sample complexity of PreTS is asymptotically tight.
Stable and consistent model alignment for language models without assuming human preference models.
problem Lack of statistical consistency in existing alignment methods.
method Relative density ratio optimization between preferred and mixture of preferred and non-preferred data distributions.
result Our approach achieves statistical consistency and stability, providing tighter convergence guarantees.
Dropping a tiny fraction of preferences can significantly alter the rankings of top LLMs.
problem Robustness of LLM ranking systems to small changes in preference data.
method A computational method based on the Bradley-Terry model to evaluate robustness.
result Top LLM rankings can be highly sensitive to the removal of a small fraction of preferences.
Paper explores limits and possibilities of aligning LLMs with human preferences.
problem Aligning LLMs with diverse human preferences to ensure fairness and informed outcomes.
method Analysis of probabilistic representation of human preferences and preservation of diverse preferences.
result LLMs can't fully align with human preferences using reward-based approaches due to Condorcet cycles, but mixed strategies are statistically possible.
Paper improves parameter estimation of continuous distributions using preference feedback.
problem Improving parameter estimation of continuous distributions.
method Preference-based M-estimators and deterministic preferences.
result Preference-based estimators achieve an estimation error scaling of O(1/n), significantly faster than sample-only methods.
Stability proved for martingale and weak transport problems.
problem Stability of martingale and weak optimal transport problems.
method Established stability through unconventional topology considering temporal structure of martingales.
result Proved stability of martingale and weak transport problems.
This work aims to separate buying preferences from merchandise commercials in fashion e-retail.
problem Difficult to infer customer preference from sales data due to implicit signals.
method Extends earlier work on explicit signals to implicit signals from user behavior.
result Derives a metric to separate buying preferences from merchandise commercials.
New methods estimate transport-growth pairs in unbalanced optimal transport.
problem Statistical guarantees for Monge-type estimation in unbalanced optimal transport remain limited.
method Developed two estimators for transport-growth pairs under different setups.
result Achieved minimax optimal rate for estimation of transport-growth pairs.
Paper investigates monotonicity issues in AI preference learning.
problem AI models may violate monotonicity when learning preferences.
method Investigates root causes of non-monotonicity in comparison-based preference learning.
result Proves local pairwise monotonicity under mild assumptions.
DOPL learns from preference feedback to solve RMAB problems.
problem Learning optimal decisions in RMAB with limited reward information.
method Direct online preference learning (DOPL) for Pref-RMAB.
result DOPL achieves sublinear regret for RMAB with preference feedback.
Direct Density Ratio Optimization aligns LLMs with human preferences without assuming specific models.
problem Statistical inconsistency in aligning LLMs with human preferences.
method Direct Density Ratio Optimization (DDRO) estimates density ratio directly.
result DDRO is statistically consistent, converging to true human preferences as data grows.
Bayesian optimization with preference learning identifies preferred solutions in multi-objective problems.
problem Optimizing multiple criteria with decision maker preferences in expensive functions.
method Bayesian optimization with interactive preference learning and active acquisition function.
result Identifies the most preferred solution with reduced interaction cost.
CPMetric learns distances between structured preferences using deep neural networks.
problem Learning distances between structured preference representations.
method Deep Siamese Networks and CP-net formalism for metric learning.
result CPDist outperforms existing approximation algorithms in accuracy and computation time.
Training models to prefer certain responses can unintentionally shift probability to harmful ones.
problem Likelihood displacement in DPO models, leading to unintended unalignment.
method Characterized and mitigated likelihood displacement using CHES score.
result Training models to prefer certain responses can unintentionally shift probability mass to harmful responses.
This work proves win rate is key to understanding preference learning.
problem Understanding preference learning from generative models.
method Analyzing preference learning methods as win rate optimization or non-WRO.
result Proves win rate is the only evaluation respecting preferences and prevalences.
IDT learns human preferences from uncertain decisions, even when humans are suboptimal.
problem Learning human preferences from uncertain and suboptimal decisions.
method Inverse decision theory (IDT) framework, statistical analysis of IDT, characterizing sample complexity.
result Learning preferences is easier when decisions are more uncertain, even if humans are suboptimal.
The paper proposes a method to infer multi-objective rewards from preferences.
problem Modeling preferences based on multiple, often competing objectives.
method Modeling priorities lexicographically and inferring multi-objective rewards from observed preferences.
result Lexicographically-ordered rewards provide a better understanding of preferences and improve policies.
New method trains neural ODEs faster with fewer layers.
problem Training neural ODEs on large datasets is computationally expensive.
method Combines optimal transport and stability regularizations.
result Significant reductions in training time with no performance loss.
The paper explores game-theoretic alignment of LLMs with human preferences, finding limitations and conditions.
problem Aligning LLMs with human preferences using game theory.
method Systematic study of payoff choices in a two-player zero-sum game for desirable alignment properties.
result Impossibility of preference matching in game-theoretic LLM alignment under standard assumptions.
Novel approach learns optimal transport using convex neural networks.
problem Learning optimal transport between distributions from samples.
method Solving a minimax optimization to learn two convex functions, representing the optimal transport map.
result The approach finds optimal transport mappings that are independent of initialization and can handle discontinuous distributions.
Introduces statistical optimal transport for probabilistic lectures.
problem No specific problem stated; focuses on introduction.
method Lecture-based introduction to statistical optimal transport.
result Provides an introduction to statistical optimal transport.
Bayesian DL model improves DCMs for better predictive and inferential performance.
problem Limited interpretability and predictive underperformance of traditional DCMs.
method Integrates deep learning with approximate Bayesian inference (SGLD).
result Improves predictive and inferential metrics in discrete choice models.
Bal-PM reduces preference labeling costs for LLMs.
problem Efficiently acquiring human feedback for preference modeling in large language models.
method Bayesian Active Learning with entropy maximization in feature space.
result Bal-PM reduces the number of required preference labels by 33% to 68%.
DPA aligns LLMs with multi-objective rewards for diverse user preferences.
problem Fine-grained control over LLMs for diverse user needs.
method Integrates multi-objective reward modeling and directional preference control.
result DPA offers better performance trade-offs and intuitive user control over LLM generation.
Study shows how optimal transport behaves in higher dimensions.
problem Characterizing optimal transport in higher dimensions with Euclidean distance.
method Investigates the small regularization limit of entropic optimal transport.
result The limiting transport plan is supported on transport rays and uniquely minimizes a relative entropy functional.