Designers of AI agents often iterate on the reward function in a trial-and-error process until they get the desired behavior, but this only guarantees good behavior in the training environment. We propose structuring this process as a series of queries asking the user to compare between different reward functions. Thus…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “reward queries”
Optimizes reward learning design for complex tasks using nonparametric methods.
problem Challenges in specifying reward functions for complex tasks.
method Models rewards and policies as nonparametric functions in RKHSs, derives risk bounds, and optimizes query design.
result Derives non-asymptotic excess risk bounds and finite sample statistical rates for reward learning.
ARL uses queries to learn rewards, focusing on cost vs. reward value.
problem How to efficiently use queries to learn rewards in reinforcement learning.
method Proposed and evaluated heuristic approaches for ARL in multi-armed bandits and MDPs.
result Challenging aspects of ARL highlighted, including intractability of value computation.
Efficient exploration improves large language model performance with fewer queries.
problem Improving large language model performance with fewer human feedback queries.
method Sequentially generates queries, fits a reward model to feedback, uses double Thompson sampling with epistemic neural network uncertainty.
result Efficient exploration enables high performance with far fewer queries.
Quantum algorithms for multi-armed bandits are explored with limited reward access.
problem Exploring quantum speed-ups in multi-armed bandit problems with limited reward information.
method Introduced new bandit models and showed query complexity equivalence with classical algorithms.
result No quadratic speed-up is possible for multi-armed bandits with limited reward access.
We seek to align agent policy with human expert behavior in a reinforcement learning (RL) setting, without any prior knowledge about dynamics, reward function, and unsafe states. There is a human expert knowing the rewards and unsafe states based on his preference and objective, but querying that human expert is expens…
Efficiently learns reward functions with fewer queries and shorter computation times.
problem Expensive data generation and labeling in robot learning.
method Batch active preference-based learning methods using determinantal point processes (DPP) and heuristic alternatives.
result Our batch active learning algorithm requires only a few queries and computes them in a short amount of time.
New method reduces regret in budgeted learning problems.
problem Decision-making with limited reward queries.
method Confidence-Budget Matching (CBM) principle.
result CBM-based algorithms perform well in adversarial settings.
We characterize learnability for stochastic noisy bandits, identifying optimal query complexities.
problem Learnability of stochastic noisy bandit models.
method Complete characterization through model class analysis and proof of optimal query complexities.
result Characterization of learnability for stochastic noisy bandit models.
Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Designcs.LG
Meta-algorithm for efficient reinforcement learning from human preferences.
problem Learning from human preference comparisons in Markov decision processes.
method Randomized exploration and experimental design for batch comparison queries.
result Meta-algorithm achieves both regret and last-iterate guarantees with minimal preference queries.
We seek to align agent behavior with a user's objectives in a reinforcement learning setting with unknown dynamics, an unknown reward function, and unknown unsafe states. The user knows the rewards and unsafe states, but querying the user is expensive. To address this challenge, we propose an algorithm that safely and …
Proposes an online model for LLM cascading with adaptive API selection.
problem Adaptive querying and selection of LLM APIs in a context-dependent environment.
method Develops a learning approach combining GMM estimation and UCB-style bounds.
result Achieves cumulative regret of over periods.
Oracle-efficient algorithms reduce combinatorial semi-bandit regret to logarithmic time.
problem Scalability issue in combinatorial semi-bandit problems due to high combinatorial optimization costs.
method Oracle-efficient frameworks that minimize oracle queries while maintaining tight regret guarantees.
result Achieved regret with oracle queries for worst-case linear rewards.
Data generation and labeling are usually an expensive part of learning for robotics. While active learning methods are commonly used to tackle the former problem, preference-based learning is a concept that attempts to solve the latter by querying users with preference questions. In this paper, we will develop a new al…
Solves TOD systems' query annotation problem without explicit annotations.
problem Training TOD systems without explicit KB query annotation.
method Reinforcement learning (RL) and pipelined approach for query prediction and system training.
result Improved RL agent with modifications for TOD tasks.
This paper optimizes slate decision systems for large action spaces.
problem Optimizing large-scale decision systems with arbitrary reward functions.
method A policy optimization framework with a novel relaxation of decision functions.
result Demonstrates the effectiveness of the proposed method on large action spaces.
A new algorithm for differential privacy in kernelized contextual bandits reduces error rate.
problem Joint differential privacy in kernelized contextual bandits.
method Proposes a novel algorithm with a specific error rate and privacy parameter dependence.
result Achieves an error rate of $\mathcal{O}\left(\sqrt{\frac{γ_T}{T}} + \frac{γ_T}{T \varepsilon}
ight)$ after queries.
Paper addresses privacy and robustness in stochastic linear bandits.
problem Stochastic linear bandits with differential privacy and adversarial robustness.
method Logarithmic batch queries, arm elimination algorithm, two privacy models.
result First algorithms providing differential privacy and adversarial robustness.
sGPO optimizes RLVR training by balancing inference FLOPs and training efficiency.
problem RLVR training allocates rollout budget without considering query difficulty.
method sGPO trades inference FLOPs for reduced training FLOPs.
result sGPO matches or exceeds baseline performance while reducing training compute by a factor of three.
ALINE optimizes Bayesian inference and data acquisition by strategically querying informative data.
problem Strategic acquisition of informative data for Bayesian inference in challenging tasks.
method Unified framework combining amortized Bayesian inference and active data acquisition using a transformer architecture trained via reinforcement learning.
result ALINE delivers both instant and accurate inference along with efficient selection of informative points.
Post-training optimizes model performance beyond base model limits.
problem Optimizing sequence prediction models beyond the base model's support.
method Policy gradient (PG) and adaptive learning rate (LR) techniques.
result Post-training with PG can achieve near-optimal performance beyond the base model's support.
Given a binary prediction problem, which performance metric should the classifier optimize? We address this question by formalizing the problem of Metric Elicitation. The goal of metric elicitation is to discover the performance metric of a practitioner, which reflects her innate rewards (costs) for correct (incorrect)…
Proposes a new theoretical framework for PbRL that requires less human feedback.
problem Lack of theoretical work capturing practical PbRL frameworks.
method Introduces a reward-agnostic PbRL framework that acquires exploratory trajectories before human feedback.
result Demonstrates improved sample complexity for learning optimal policies in linear and low-rank MDPs.
The paper enhances preference learning by incorporating response time data.
problem Lack of temporal information in user decision-making for reward model learning.
method Integrates response time alongside binary choice data using the EZ model and Neyman-orthogonal loss functions.
result Response time-augmented approach reduces error rates from exponential to polynomial scaling, improving sample efficiency.
Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysiscs.LG
COCOA improves credit assignment in reinforcement learning by measuring contributions to rewards.
problem Improving sample efficiency in reinforcement learning through better credit assignment methods.
method Counterfactual Contribution Analysis (COCOA) for precise credit assignment.
result COCOA achieves lower bias and variance compared to Hindsight Credit Assignment (HCA), improving reinforcement learning performance.
When estimating the relevancy between a query and a document, ranking models largely neglect the mutual information among documents. A common wisdom is that if two documents are similar in terms of the same query, they are more likely to have similar relevance score. To mitigate this problem, in this paper, we propose …
Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inferencecs.LG
Paper develops methods to optimize policies directly from human feedback without reward inference.
problem Challenges in RLHF, including reward model overfitting and distribution shift.
method Develops two algorithms for RLHF without reward inference, using zeroth-order gradient approximators.
result Establishes polynomial convergence rates and outperforms existing methods in numerical experiments.
Generative Flow Networks use submodular upper bounds to generate more data.
problem Generating data from unknown, complex reward functions efficiently.
method Introduce submodular upper bounds to estimate reward, use Optimism in the Face of Uncertainty principle to train GFNs.
result SUBo-GFN generates significantly more data than classical GFNs.
Unified framework controls false discovery rate in bandit multiple testing.
problem Designing adaptive algorithms to identify true discoveries in multiple hypothesis testing.
method Unified modular framework using e-processes for FDR control in arbitrary settings.
result Unified framework ensures FDR control for dependent and simultaneous arm queries.
No-regret optimization for time-varying functions using uncertainty injection.
problem Optimizing time-varying functions with no-regret in bandit feedback.
method W-SparQ-GP-UCB, incorporating uncertainty injection and additional queries.
result Achieves no-regret with a vanishing number of additional queries per iteration.
New method for efficient online exploration in RLHF reduces regret.
problem Efficiently collecting new preference data in RLHF to refine reward model and policy.
method Proposes a new exploration scheme that directs preference queries toward reducing uncertainty in reward differences most relevant to policy improvement.
result Establishes regret bounds of order for online RLHF, with polynomial scaling in all model parameters.
New RLHF framework handles general preference oracles without reward functions.
problem Handling general preference oracles without assuming a reward function.
method Developed a minimax game between two LLMs for RLHF under a general preference oracle, focusing on KL-regularized preference.
result Proposed algorithms for efficient offline and online RLHF learning.
This work improves online fine-tuning of diffusion models for specific properties.
problem Efficiently fine-tuning diffusion models to maximize specific properties.
method A novel reinforcement learning procedure that efficiently explores feasible samples.
result The method provides a regret guarantee and empirical validation across multiple domains.
GAMBITTS uses GenAI for adaptive interventions, improving decision-making.
problem Adaptive interventions with GenAI-generated content.
method Generator-mediated bandit-Thompson sampling (GAMBITTS).
result GAMBITTS outperforms standard bandit methods in mobile health interventions.
The combination of deep neural network models and reinforcement learning algorithms can make it possible to learn policies for robotic behaviors that directly read in raw sensory inputs, such as camera images, effectively subsuming both estimation and control into one model. However, real-world applications of reinforc…
MAXMINLCB optimizes unknown target functions with preference feedback using a Stackelberg game approach.
problem Optimizing unknown target functions with pairwise comparisons and human feedback.
method MAXMINLCB, a zero-sum Stackelberg game, balances exploration and exploitation.
result MAXMINLCB consistently outperforms existing algorithms with a rate-optimal regret guarantee.
Study on the limits of bandit learning, showing hardness and limitations.
problem Understanding the learnability of bandit learning under arbitrary reward functions.
method Investigation into which classes of reward functions are learnable and how they can be learned.
result No combinatorial dimension can characterize bandit learnability, and computational hardness is inherent.
Dual active learning improves RLHF by selecting optimal conversations and teachers.
problem Efficiently aligning LLMs with human preferences using RLHF from feedback.
method Offline RL for conversation and teacher selection, dual active reward learning, pessimistic RL.
result The proposed algorithm achieves minimal generalized variance and outperforms state-of-the-arts.
A new adversarial attack method using structured search and contextual bandits.
problem Black-box adversarial attacks on deep learning models.
method Structured search space and Bayesian optimization for contextual bandits.
result Achieves state-of-the-art success rates and query efficiencies.
Active IRL selects optimal human demonstrations for learning AI preferences.
problem Costly human demonstrations in IRL for autonomous systems.
method Information-theoretic acquisition function for selecting informative human demonstrations.
result Efficiently reduces human effort in learning AI preferences.
We consider differentially private algorithms for reinforcement learning in continuous spaces, such that neighboring reward functions are indistinguishable. This protects the reward information from being exploited by methods such as inverse reinforcement learning. Existing studies that guarantee differential privacy a…
The emergence of structured databases for Question Answering (QA) systems has led to developing methods, in which the problem of learning the correct answer efficiently is based on a linking task between the constituents of the question and the corresponding entries in the database. As a result, parsing the questions i…
TensorPlan shows an exponential lower bound for planning in MDPs with linearly realizable value functions.
problem Finding an exponential lower bound for planning in MDPs with linearly realizable value functions.
method TensorPlan and a few action lower bound approach.
result An exponentially large lower bound is shown for planning in MDPs with linearly realizable value functions.
Order-Optimal Regret in Distributed Kernel Bandits using Uniform Sampling with Shared Randomnesscs.LG
Algorithm reduces regret in distributed kernel bandits with shared randomness.
problem Minimizing regret in collaborative function maximization.
method Uniform exploration at local agents and shared randomness with central server.
result Achieves optimal regret order with sublinear communication cost.
Inverse reinforcement learning (IRL) infers a reward function from demonstrations, allowing for policy improvement and generalization. However, despite much recent interest in IRL, little work has been done to understand the minimum set of demonstrations needed to teach a specific sequential decision-making task. We fo…
Meta-AAD uses deep reinforcement learning to improve anomaly detection by selecting the most informative instances.
problem High false-positive rate in anomaly detection, especially in high-stake applications.
method Meta-AAD leverages deep reinforcement learning to train a meta-policy for query selection, optimizing the number of discovered anomalies.
result Meta-AAD significantly outperforms state-of-the-art re-ranking strategies and unsupervised baselines on 24 benchmark datasets.
Sufficient supervised information is crucial for any machine learning models to boost performance. However, labeling data is expensive and sometimes difficult to obtain. Active learning is an approach to acquire annotations for data from a human oracle by selecting informative samples with a high probability to enhance…
Paper tackles non-stationary kernelized bandits with near-optimal algorithm.
problem Minimizing regret in a time-varying reward function.
method Near-optimal algorithm with a novel restarting phased elimination with random permutation (R-PERP).
result Regret upper bound matches the lower bound, making the algorithm near-optimal.