New insights into cascade feedback linearization of control systems.
problem Obtaining a cascade feedback linearization for invariant control systems.
method Introducing truncated versions of operators from the calculus of variations to prove new theorems.
result Established new geometry and foundational theorems for future work.
New method learns from either positive or negative feedback alone.
problem Limited applicability of existing preference optimization methods in scenarios with only unpaired feedback.
method Decouples learning from positive and negative feedback, using expectation-maximization (EM) to optimize probability of positive outcomes and explicitly incorporate negative examples.
result Stable learning from negative feedback alone demonstrated.
User preferences for items can be inferred from either explicit feedback, such as item ratings, or implicit feedback, such as rental histories. Research in collaborative filtering has concentrated on explicit feedback, resulting in the development of accurate and scalable models. However, since explicit feedback is oft…
This paper improves image retrieval accuracy through novel relevance feedback methods.
problem Improving image retrieval accuracy in Content-Based Image Retrieval (CBIR).
method Novel addition to feature re-weighting and classification techniques, focusing on 0-th iteration improvement.
result Significantly improved retrieval accuracy from relevance feedback.
Recommender systems recommend items more accurately by analyzing users' potential interest on different brands' items. In conjunction with users' rating similarity, the presence of users' implicit feedbacks like clicking items, viewing items specifications, watching videos etc. have been proved to be helpful for learni…
Study evaluates new models using human feedback from another model.
problem Evaluate a new model using human feedback collected for another model.
method Formalize problem, propose model-based and model-free estimators, analyze unbiasedness, and empirically evaluate.
result Proposed estimators can predict absolute values, rank, and optimize evaluated policies.
Develops a new model for RLHF accounting for partially observed states and intermediate feedback.
problem Lack of models for partially observed states and intermediate feedback in RLHF.
method PORRL model with cardinal and dueling feedback methods.
result Demonstrates improved learning and alignment with new model-based and model-free methods.
Classifier learns to ignore unreliable feedback from end users.
problem Improving classifier performance by filtering unreliable feedback.
method Modeling end users as autonomous agents, periodically retraining classifier with filtered feedback.
result Classifier can identify and filter out unreliable feedback, improving performance.
Paper characterizes minimax regret rates for online ranking with top-k feedback.
problem Analyzing online ranking with partial feedback.
method Developed techniques from partial monitoring to characterize minimax regret rates.
result Full characterization of minimax regret rates for Precision@n.
Study on sample complexity for pure exploration in feedback graph settings.
problem Sample complexity of pure exploration in online learning with feedback graphs.
method Derive instance-specific lower bounds and present asymptotically optimal algorithm TaS-FG.
result TaS-FG is asymptotically optimal and efficient across different graph configurations.
One-bit feedback suffices for a bandit problem's optimal strategy.
problem Optimal strategy for multi-armed bandit problem with limited feedback.
method Coding and decoding schemes for one-bit feedback to mimic full-reward feedback.
result Regret ratio approaches 1 with one-bit feedback.
Advocates a local feedback approach for RL in unknown systems.
problem Finding optimal feedback laws in unknown nonlinear dynamical systems.
method Searches over a local feedback representation consisting of an open-loop sequence and an optimal linear feedback law.
result Results in highly efficient training and superior performance compared to global methods.
Study how communication and feedback graphs affect learning outcomes.
problem Understanding the impact of feedback graphs on cooperative online learning.
method Analyzed network regret in terms of the independence number of the strong product of communication and feedback graphs.
result Proved bounds for network regret and demonstrated the non-improvable nature of positive results in pathological cases.
New method uses correlated auxiliary feedback to reduce regret in parameterized bandits.
problem Reducing regret in parameterized bandits with correlated auxiliary feedback.
method Develops a reward estimator using auxiliary feedback with tight confidence bounds.
result Shows significant reduction in regret compared to standard methods.
Test for linearizing 2-input systems with 2D feedback.
problem Linearizability of two-input systems by feedback.
method Algorithmic test for 2D endogenous feedback.
result Systematic derivation of flat outputs.
The problem of feedback equivalence for control systems is considered. An algebra of differential invariants and criteria for the feedback equivalence for regular control systems are found.
Reduces user feedback needed for accurate recommender systems.
problem Limited user feedback in recommender systems.
method Partial Bandit and Semi-Bandit approach for efficient user feedback retrieval.
result Similar global accuracy and learning efficiency with reduced feedback.
We present a study on reinforcement learning (RL) from human bandit feedback for sequence-to-sequence learning, exemplified by the task of bandit neural machine translation (NMT). We investigate the reliability of human bandit feedback, and analyze the influence of reliability on the learnability of a reward estimator,…
New algorithm for recommending best arms with aggregated feedback.
problem Finding the best arm under aggregated feedback when precise rewards are unavailable.
method Gaussian Process Optimistic Optimisation (GPOO) algorithm with adaptive tree construction.
result The proposed algorithm achieves new simple regret bounds with aggregated feedback.
Study of reinforcement learning with additional feedback observations.
problem Episodic reinforcement learning in Markov decision processes with feedback observations.
method Formalization of feedback graph, model-based algorithms leveraging feedback, regret bound analysis.
result Regret bound depends only on the size of the maximum acyclic subgraph of the feedback graph.
CausalRM models rewards from user feedback, overcoming noise and bias.
problem Aligning language models with user preferences from noisy, biased feedback.
method Causal-theoretic reward modeling framework addressing noise and bias in observational feedback.
result CausalRM learns accurate reward signals from noisy and biased observational feedback.
Paper tackles noisy bandit feedback for multiclass classification.
problem Learning multiclass classifier with corrupted feedback.
method Proposes an unbiased estimator technique to estimate noise rates and an end-to-end framework.
result Algorithm achieves mistake bounds of O ( T ) O(\sqrt{T}) O ( T ) in high noise and O ( T i c e f r a c 23 ) O(T^{
icefrac{2}{3}}) O ( T i ce f r a c 2 3 ) in worst case. We propose a generalization of the best arm identification problem in stochastic multi-armed bandits (MAB) to the setting where every pull of an arm is associated with delayed feedback. The delay in feedback increases the effective sample complexity of standard algorithms, but can be offset if we have access to partial…
Easy conditions found for simplifying complex systems.
problem Linearizing complex two-input systems.
method Endogenous dynamic feedback with a dimension of at most two.
result Necessary and sufficient conditions for linearizability.
Active learning framework for optimizing human preferences in reinforcement learning.
problem Selecting most informative feedback for training models of human preferences.
method Proposes an active learning framework to collect preferential feedback online or offline.
result Errors in DPO logit estimates diminish with more feedback.
Paper studies CLO with partial feedback, improving decision-making in uncertain contexts.
problem Improving decision-making in contexts with uncertain cost coefficients using partial feedback.
method Unified class of offline learning algorithms for CLO with different types of feedback, using IERM framework.
result Fast-rate regret bound for IERM with partial feedback and misspecified model classes.
We use online convex optimization (OCO) for setpoint tracking with uncertain, flexible loads. We consider full feedback from the loads, bandit feedback, and two intermediate types of feedback: partial bandit where a subset of the loads are individually observed and the rest are observed in aggregate, and Bernoulli feed…
CAFL breaks feedback loops in recommender systems using causal inference.
problem Feedback loops in recommender systems compromise recommendation quality and homogenize user behavior.
method Causal Adjustment for Feedback Loops (CAFL) algorithm that breaks feedback loops using causal inference.
result CAFL improves recommendation quality compared to prior correction methods.
We consider combinatorial online learning with subset choices when only relative feedback information from subsets is available, instead of bandit or semi-bandit feedback which is absolute. Specifically, we study two regret minimisation problems over subsets of a finite ground set [ n ] [n] [ n ] , with subset-wise relative prefe…
Unified framework for expert selection with bandit and lower-bound feedback.
problem Selecting the best expert in scenarios with bandit feedback and lower-bound information.
method Introduces a new feedback model combining bandit and lower-bound information, proving optimal regret bounds for modified Exp3 algorithms.
result Optimal regret bounds for modified Exp3 algorithms, generalizing both bandit and full-information settings.
GAIF enhances online multiple testing with feedback, improving statistical power.
problem Sequential online multiple testing with delayed feedback.
method GAIF framework using dynamic threshold adjustment and feedback-driven model selection.
result Improves statistical power through feedback-driven model selection.
New algorithms handle online prediction with bandit and delayed feedback, improving regret bounds.
problem Achieving finite bounds on surrogate regret with limited feedback.
method Proposed algorithms for bandit and delayed feedback, including inverse-weighted gradient and pseudo-inverse matrix estimators.
result Achieved improved surrogate regret bounds of O ( K T ) O(\sqrt{KT}) O ( K T ) and O ( T 2 / 3 ) O(T^{2/3}) O ( T 2/3 ) . New method provides fine-grained feedback on interactive student programs.
problem Time-consuming manual grading of interactive student programs.
method Meta-exploration approach using reinforcement learning.
result 94.3% accuracy in providing fine-grained feedback.
The paper proposes a new method for learning reward models from ordinal feedback, improving upon binary feedback.
problem Learning reward models from human preferences using binary feedback discards useful samples and loses fine-grained information.
method The paper introduces a framework for learning reward models under ordinal feedback, generalizing the Bradley-Terry model.
result Ordinal feedback reduces the Rademacher complexity compared to binary feedback, leading to better reward learning.
We study an online learning framework introduced by Mannor and Shamir (2011) in which the feedback is specified by a graph, in a setting where the graph may vary from round to round and is \emph{never fully revealed} to the learner. We show a large gap between the adversarial and the stochastic cases. In the adversaria…
New algorithms ensure fair selection in combinatorial semi-bandit with unrestricted delays.
problem Fair selection in stochastic combinatorial semi-bandit with delayed feedback.
method Introduced merit-based fairness constraints and new bandit algorithms for reward and fairness.
result Achieved sublinear expected reward and fairness regrets with dependence on delay distribution quantiles.
The paper tackles combinatorial pure exploration with various feedback structures and proposes efficient algorithms.
problem Identifying the optimal action in a combinatorial space with limited feedback and nonlinear rewards.
method Designs polynomial-time adaptive algorithms for CPE-BL and CPE-PL, providing sample complexity analyses.
result The proposed algorithms achieve sample complexity close to lower bounds and outperform existing methods.
New method for online learning IC models with node-level feedback.
problem Learning IC models with node-level feedback in social networks.
method Detailed analysis and online algorithm with O ( T ) \mathcal{O}( \sqrt{T}) O ( T ) cumulative regret. result First confidence-region result and online algorithm for IC models with node-level feedback.
New Q-learning algorithms reduce regret in inventory control problems.
problem Efficiently learning optimal policies in inventory control problems with limited feedback.
method Proposed Elimination-Based Half-Q-Learning (HQL) and Full-Q-Learning (FQL) algorithms with theoretical regret bounds.
result HQL incurs i l d e O ( H 3 T ) ilde{\mathcal{O}}(H^3\sqrt{ T}) i l d e O ( H 3 T ) regret, FQL incurs i l d e O ( H 2 T ) ilde{\mathcal{O}}(H^2\sqrt{ T}) i l d e O ( H 2 T ) regret, independent of state and action space sizes. We present and study models of adversarial online learning where the feedback observed by the learner is noisy, and the feedback is either full information feedback or bandit feedback. Specifically, we consider binary losses xored with the noise, which is a Bernoulli random variable. We consider both a constant noise r…
Capacity-Constrained Online Convex Optimization with Delayed Feedback
problem Online learning with delayed feedback under a hard capacity constraint
method Reduction to a delayed and weighted OCO problem using a scheduler
result First regret guarantees for capacity-constrained OCO under convex and strongly convex losses
The problem of local feedback equivalence for 1-dimensional control systems of the 1-st order is considered. The algebra of differential invariants and criteria for the feedback equivalence for regular control systems are found.
Reinforcement learning with trajectory feedback instead of state-action rewards.
problem Frequent feedback not available in practice.
method Extended reinforcement learning algorithms using trajectory feedback for known and unknown transition models.
result Hybrid optimistic-Thompson Sampling algorithm for unknown transition models.
Interactive Machine Learning is concerned with creating systems that operate in environments alongside humans to achieve a task. A typical use is to extend or amplify the capabilities of a human in cognitive or physical ways, requiring the machine to adapt to the users' intentions and preferences. Often, this takes the…
Efficiently identifies good policies by choosing contexts for human feedback.
problem Efficiently identifying good policies in applications with high feedback costs.
method Introduces offline contextual dueling bandit setting and an upper-confidence-bound style algorithm.
result Proves a regret bound and shows superior performance over uniformly sampled contexts.
Neural algorithms optimize arm selection with human preference feedback for complex reward functions.
problem Optimizing arm selection with noisy human preference feedback for complex, non-linear reward functions.
method Neural network to estimate reward function using preference feedback, upper confidence bound and Thompson sampling algorithms.
result Sub-linear regret guarantees for efficient arm selection in contextual dueling bandits.
Sayer uses implicit feedback to optimize system policies.
problem Leveraging implicit feedback to improve system policies is difficult due to bias and incompleteness.
method Sayer combines randomized exploration and unbiased counterfactual estimators to evaluate and train new policies using implicit feedback.
result Sayer can accurately evaluate and train new policies that outperform existing ones.
Hybrid approach combines user feedback and machine learning for predicting user satisfaction.
problem Measuring user satisfaction in large-scale conversational agent systems.
method Fusion of explicit user feedback and predictions from two machine-learned models trained on different data types.
result Hybrid approach significantly improves user satisfaction predictions.