Enhances AI models with human feedback for noisy data.
problem Improving AI model alignment with human feedback in noisy environments.
method Two-stage SL+LHF framework connecting machine learning with human feedback.
result The LNCA ratio identifies conditions for SL+LHF superiority over pure SL.
New oracle uses uncertainty for active classification with noisy feedback.
problem Improving query complexity in interactive binary classifier learning.
method Proposes a new pairwise comparison oracle that considers uncertainty and an adaptive labeling algorithm.
result Demonstrates improved performance and efficiency compared to existing methods.
Proposes a TS approach for Bayesian optimization with preferential feedback.
problem Optimizing with preference feedback in complex applications.
method Uses Thompson Sampling with a dueling kernel and anchor invariance.
result Performance matches standard TS for scalar feedback in finite time.
New algorithm learns human preferences from few comparisons efficiently.
problem Learning human preferences from limited comparison feedback.
method Formulated as D-optimal design for Plackett-Luce model, solved using randomized Frank-Wolfe algorithm.
result Proposed algorithm efficiently solves D-optimal design problem for Plackett-Luce objective.
Binary feedback outperforms ordinal comparisons in ranking recovery.
problem Challenges the conventional wisdom that ordinal comparisons offer richer information.
method Proposes a parametric framework for modeling ordinal paired comparisons, binarizing ordinal data, and proving faster convergence rates for binary comparisons.
result Binarizing ordinal data significantly improves ranking recovery accuracy and exhibits a substantial performance gap.
The paper analyzes RLHF with human feedback and provides convergence results for MLE and pessimistic MLE.
problem Improving RLHF with human feedback from pairwise or K K K -wise comparisons. method Theoretical framework for RLHF with convergence analysis of MLE and pessimistic MLE.
result MLE fails but pessimistic MLE provides improved policies under certain coverage assumptions.
Improved sample efficiency in preference-based RL with multiple comparisons.
problem Sample inefficiency in preference-based reinforcement learning with pairwise comparisons.
method Proposes M-AUPO, an algorithm that selects multiple actions by maximizing average uncertainty within subsets.
result Achieves a suboptimality gap of $O\left( \frac{d}{T} \sqrt{ \sum_{t=1}^T \frac{1}{|S_t|}}
ight)$ , improving performance with larger subsets.
Study controlled contagion with state-dependent killing, proving a comparison principle.
problem Analyzing controlled McKean--Vlasov contagion with state-dependent killing.
method Proof of a comparison principle using Wasserstein smooth-gauge comparison and killing-jump absorption estimates.
result Established a comparison principle for the two-population killed-particle HJB.
In supervised learning, we typically leverage a fully labeled dataset to design methods for function estimation or prediction. In many practical situations, we are able to obtain alternative feedback, possibly at a low cost. A broad goal is to understand the usefulness of, and to design algorithms to exploit, this alte…
This thesis tackles learning reward functions from human comparative feedback.
problem Designing reward functions for complex tasks is challenging and humans often provide suboptimal demonstrations.
method Proposes learning reward functions from comparative feedback (pairwise comparisons, best-of-many choices, rankings, scaled comparisons) and active learning techniques.
result Demonstrates the effectiveness of learning reward functions from comparative feedback in various domains.
New framework PBBO optimizes latent functions with preferential feedback.
problem Optimizing latent functions with parallel preferential feedback.
method Gaussian process model with a likelihood for parallel comparisons.
result Framework PBBO generalizes and expands Bayesian optimization.
Meta-algorithm for efficient reinforcement learning from human preferences.
problem Learning from human preference comparisons in Markov decision processes.
method Randomized exploration and experimental design for batch comparison queries.
result Meta-algorithm achieves both regret and last-iterate guarantees with minimal preference queries.
We investigate contextual online learning with nonparametric (Lipschitz) comparison classes under different assumptions on losses and feedback information. For full information feedback and Lipschitz losses, we design the first explicit algorithm achieving the minimax regret rate (up to log factors). In a partial feedb…
New algorithm for identifying Condorcet team in noisy comparisons.
problem Online learning with noisy comparisons of teams.
method Formalized dueling teams problem, developed algorithms for stochastic and deterministic settings.
result Identifies Condorcet winning team with reduced number of duels.
Proposes a method for ranking items across multiple aspects based on user feedback.
problem No principled solution exists for generating multiple item rankings over different aspects.
method Developed a directional multi-aspect ranking criterion using probabilistic multivariate tensor factorization.
result Demonstrated effectiveness of the proposed method through comprehensive experiments on real datasets.
We present a simple uniqueness argument for a collection of McKean-Vlasov problems that have seen recent interest. Our first result shows that, in the weak feedback regime, there is global uniqueness for a very general class of random drivers. By weak feedback we mean the case where the contagion parameters are small e…
Study analyzes feedback complexity for sparse feature retrieval in deep networks.
problem Learning sparse superposed features with feedback.
method Analysis of feedback complexity in sparse settings, including triplet comparisons.
result Establishes tight bounds and strong upper bounds for feature recovery.
New algorithm for linear bandits learns from feature feedback, reducing regret.
problem Linear bandit problem with feature feedback.
method Developed new theory and algorithms for linear bandits with feature feedback.
result Achieves regret scaling like k T k\sqrt{T} k T , improving over traditional linear bandits. Study preference-based reinforcement learning in episodic kernel MDPs.
problem Learning from episodic human preferences in reinforcement learning.
method Developed preference-based value estimation and confidence sets for kernel-based MDPs.
result Proved high-probability regret bounds that converge to optimal policy value.
A new dual test for forward-flatness simplifies computations.
problem Checking forward-flatness in discrete-time systems.
method A unique sequence of integrable codistributions.
result Computational efficiency and comparison with dynamic feedback linearization.
Unified LP framework for offline reward learning from human demonstrations and feedback.
problem Reward learning from human demonstrations and feedback with robustness and sample efficiency.
method A novel linear programming framework for offline reward learning.
result Unified LP framework achieves better performance compared to MLE.
Proposes a new theoretical framework for PbRL that requires less human feedback.
problem Lack of theoretical work capturing practical PbRL frameworks.
method Introduces a reward-agnostic PbRL framework that acquires exploratory trajectories before human feedback.
result Demonstrates improved sample complexity for learning optimal policies in linear and low-rank MDPs.
New algorithms improve on bandit feedback in matrix games with unknown payoff matrices.
problem Improving performance in matrix games with unknown payoff matrices and bandit feedback.
method Regret analyses of variants of UCB and K-learning.
result New algorithms achieve lower regret compared to adversarial bandit algorithms.
PDO optimizes LLM prompts without labels, improving performance.
problem Optimizing prompts for LLMs without access to labeled data.
method Pairwise preference feedback, dueling bandits, Thompson Sampling, mutation.
result PDO identifies stronger prompts than label-free methods.
RANDomized-exploration policy Optimization via Multiple Importance Sampling with Truncation (RANDOMIST) for PO with mediator feedback.
problem Policy Optimization in continuous control tasks.
method RANDomized-exploration policy Optimization via Multiple Importance Sampling with Truncation (RANDOMIST) for regret minimization in PO.
result Achieving constant regret under certain circumstances in PO with mediator feedback.
PSEUDo learns patterns in multivariate time series with locality-sensitive hashing and relevance feedback.
problem Efficient pattern detection in large, multi-track sequential data with high variance and lack of ground truth.
method Query-aware locality-sensitive hashing for feature learning, sub-linear training and inference time.
result PSEUDo achieves sub-linear time efficiency for pattern modeling and comparison of 10,000 multivariate time series.
This work compares human feedback methods for reward learning in bandits.
problem Understanding how human feedback affects the performance of reward learning methods.
method Theoretical comparison of human feedback approaches in offline contextual bandits.
result Human bias and uncertainty in feedback modeling impact the theoretical guarantees of reward learning methods.
MAXMINLCB optimizes unknown target functions with preference feedback using a Stackelberg game approach.
problem Optimizing unknown target functions with pairwise comparisons and human feedback.
method MAXMINLCB, a zero-sum Stackelberg game, balances exploration and exploitation.
result MAXMINLCB consistently outperforms existing algorithms with a rate-optimal regret guarantee.
Simple algorithms for Dueling Bandits with improved regret bounds.
problem Optimizing decisions in a comparison-based environment with unknown preference gaps.
method Presented simple algorithms with specific regret bounds.
result Regret bounds of order O(T^rho ) with 1/2 <= rho <= 3/4, independent of preference gaps.
New ranking algorithms are continually being developed and refined, necessitating the development of efficient methods for evaluating these rankers. Online ranker evaluation focuses on the challenge of efficiently determining, from implicit user feedback, which ranker out of a finite set of rankers is the best. Online …
Method tackles uncertainty in reward models for LLMs from heterogeneous human feedback.
problem Uncertainty in reward models for LLMs from heterogeneous human feedback.
method Heterogeneous preference framework and alternating gradient descent algorithm.
result Established theoretical guarantees for estimator convergence and asymptotic distribution.
RL-LOW algorithm achieves exponential simple regret in offline RLHF with pairwise comparisons.
problem Offline reinforcement learning from human feedback with pairwise comparisons.
method Proposes RL-LOW algorithm to minimize simple regret with exponential convergence.
result Achieves an exponential form of simple regret of \(\exp ( - Ω(n/H) )\).
SetRank tackles collaborative ranking from implicit feedback using setwise Bayesian approach.
problem Challenges in pairwise and listwise approaches for implicit feedback.
method SetRank is a novel setwise Bayesian approach that accommodates implicit feedback characteristics.
result SetRank outperforms state-of-the-art baselines on real-world datasets.
Given a binary prediction problem, which performance metric should the classifier optimize? We address this question by formalizing the problem of Metric Elicitation. The goal of metric elicitation is to discover the performance metric of a practitioner, which reflects her innate rewards (costs) for correct (incorrect)…
Study on deep neural networks for reward modeling with pairwise comparison data.
problem Reward modeling with deep neural networks in non-parametric settings.
method Established a non-asymptotic regret bound for deep reward estimators, introduced a margin-type condition.
result Improved regret bound for deep reward estimators, highlighting the importance of clear human beliefs.
We study the K K K -armed dueling bandit problem, a variation of the standard stochastic bandit problem where the feedback is limited to relative comparisons of a pair of arms. We introduce a tight asymptotic regret lower bound that is based on the information divergence. An algorithm that is inspired by the Deterministic…
Paper reviews methods for learning from bandit feedback in recommender systems.
problem Learning from historical data with unknown rewards.
method Counterfactual Risk Minimisation (CRM) methods using importance sampling and variance reduction.
result Comparison of different off-policy estimators' performance.
LLM trading agents show risk feedback can improve alignment without fine-tuning.
problem Aligning LLM trading agents with financial risk.
method TradeArena testbed, risk reports, execution simulation, memory replay.
result Risk feedback can improve alignment without fine-tuning, but not universally.
ESM-CNN uses error feedback to build a random CNN for time series forecasting.
problem Improving time series forecasting accuracy with CNNs.
method Incrementally adding random filters and neurons to adaptively construct a CNN.
result ESM-CNN outperforms state-of-the-art models in prediction accuracy and efficiency.
Modulo-SK outperforms Deepcode in error probability and feedback rounds.
problem Improving communication over noisy channels with deep learning.
method Combining classical SK scheme with modulo-arithmetic for deterministic feedback.
result Modulo-SK requires less feedback SNR and fewer rounds than Deepcode.
We study the K-armed dueling bandit problem, a variation of the standard stochastic bandit problem where the feedback is limited to relative comparisons of a pair of arms. The hardness of recommending Copeland winners, the arms that beat the greatest number of other arms, is characterized by deriving an asymptotic regr…
There is increasing interest in learning algorithms that involve interaction between human and machine. Comparison-based queries are among the most natural ways to get feedback from humans. A challenge in designing comparison-based interactive learning algorithms is coping with noisy answers. The most common fix is to …
SIBRE boosts reinforcement learning convergence by rewarding improvement over past performance.
problem Improving the rate of convergence in reinforcement learning.
method SIBRE is a reward shaping approach that rewards improvement over the agent's own past performance.
result SIBRE converges faster and more stably to the optimal policy compared to baseline RL algorithms.
New algorithms for batched dueling bandits with improved regret bounds.
problem Batched dueling bandits with noisy pairwise comparisons.
method Developed algorithms for two settings: Condorcet winner and strong stochastic transitivity.
result Regret bounds match sequential bounds using only a logarithmic number of batches.
DeepCMC compresses CSI for massive MIMO systems, reducing overhead and improving performance.
problem High CSI overhead in massive MIMO systems limits spectral efficiency.
method Deep learning-based fully convolutional neural network with residual layers and entropy coding.
result DeepCMC outperforms state-of-the-art schemes in CSI reconstruction quality for the same compression rate.
Paper formulates EnKF as optimal transport problem for unique control law.
problem Unique control law for EnKF algorithms.
method Formulated as optimal transportation problem, derived explicit control law.
result Mean squared error converges to zero with finite particles.
In machine learning, the notion of multi-armed bandits refers to a class of online learning problems, in which an agent is supposed to simultaneously explore and exploit a given set of choice alternatives in the course of a sequential decision process. In the standard setting, the agent learns from stochastic feedback …
We introduce efficient algorithms which achieve nearly optimal regrets for the problem of stochastic online shortest path routing with end-to-end feedback. The setting is a natural application of the combinatorial stochastic bandits problem, a special case of the linear stochastic bandits problem. We show how the diffi…