Designers of AI agents often iterate on the reward function in a trial-and-error process until they get the desired behavior, but this only guarantees good behavior in the training environment. We propose structuring this process as a series of queries asking the user to compare between different reward functions. Thus…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “reward design”
Optimizes reward learning design for complex tasks using nonparametric methods.
problem Challenges in specifying reward functions for complex tasks.
method Models rewards and policies as nonparametric functions in RKHSs, derives risk bounds, and optimizes query design.
result Derives non-asymptotic excess risk bounds and finite sample statistical rates for reward learning.
A novel framework refines diffusion models iteratively for better downstream reward optimization.
problem Optimizing reward functions during inference of diffusion models.
method Iterative refinement process with noising and reward-guided denoising steps.
result Superior empirical performance in protein and DNA design.
BRAID fine-tunes diffusion models to optimize reward models in offline scenarios.
problem Combining generative modeling and model-based optimization in offline scenarios.
method Conservative fine-tuning of diffusion models using RL to optimize reward models.
result BRAID outperforms existing methods in offline data, avoiding invalid designs.
In many sequential decision making tasks, it is challenging to design reward functions that help an RL agent efficiently learn behavior that is considered good by the agent designer. A number of different formulations of the reward-design problem, or close variants thereof, have been proposed in the literature. In this…
Study optimal reward schemes for inducing desired player performance in risky contests.
problem Designing optimal rewards to encourage desired performance levels in risky contests.
method Analyzed the optimal reward schemes for inducing average and specific rank performance.
result Optimal reward schemes can have surprising shapes, not just related to inequality.
This work characterizes reward function partial identifiability and its impact on policy optimization.
problem Reward function partial identifiability in complex tasks.
method Formal characterisation of partial identifiability using various reward learning data sources.
result Unified framework for comparing data sources and downstream tasks by their invariances.
Paper proposes operator deep Q-learning for quick reward adaptation.
problem Standard RL can only handle one reward function and struggles with unseen rewards.
method Develops operator neural networks to map reward functions to value functions.
result Operator deep Q-learning can quickly adapt to new reward functions.
New algorithm finds optimal policies without knowing reward functions.
problem Reward-agnostic exploration in reinforcement learning.
method Designs an algorithm that explores without reward information, achieving minimax optimality.
result Achieves provable minimax optimality in finding optimal policies for multiple reward functions.
New algorithm optimally identifies best arm in both stochastic and adversarial settings.
problem Best arm identification in stochastic and adversarial reward scenarios.
method Parameter-free algorithm designed to be robust to adversarial rewards and optimal in stochastic problems.
result Algorithm's error rate matches optimal bounds in stochastic problems and is robust to adversarial rewards.
Boosted GFlowNets improve exploration by sequentially training GFlowNets with residual rewards.
problem GFlowNets struggle to evenly explore reward landscapes, leading to poor coverage of high-reward areas.
method Sequential training of an ensemble of GFlowNets, each optimizing a residual reward.
result Boosted GFlowNets achieve better exploration and sample diversity on multimodal benchmarks and peptide design tasks.
The paper addresses human-like decision-making in multi-agent systems using bounded risk-sensitive Markov Games.
problem Modeling human-like decision-making in multi-agent systems with risk-seeking and loss-aversion behaviors.
method Forward policy design and inverse reward learning with iterative reasoning and cumulative prospect theory.
result The proposed algorithms demonstrate both risk-averse and risk-seeking behaviors in multi-agent systems.
AutoDIME automates design of multi-agent environments for RL.
problem Designing multi-agent environments for reinforcement learning is challenging.
method Developed intrinsic teacher rewards for multi-agent settings and evaluated them in various tasks.
result Value disagreement was found to be most consistent and effective across tasks.
We propose a bandit algorithm that explores by randomizing its history of rewards. Specifically, it pulls the arm with the highest mean reward in a non-parametric bootstrap sample of its history with pseudo rewards. We design the pseudo rewards such that the bootstrap mean is optimistic with a sufficiently high probabi…
Recent reinforcement learning (RL) approaches have shown strong performance in complex domains such as Atari games, but are often highly sample inefficient. A common approach to reduce interaction time with the environment is to use reward shaping, which involves carefully designing reward functions that provide the ag…
Reinforcement learning algorithms rely on carefully engineering environment rewards that are extrinsic to the agent. However, annotating each environment with hand-designed, dense rewards is not scalable, motivating the need for developing reward functions that are intrinsic to the agent. Curiosity is a type of intrins…
Paper designs a bandit algorithm without reward distribution info.
problem Designing bandit algorithms without reward distribution info.
method Alternates between greedy rule and forced exploration.
result Achieves substantial regret upper bounds.
This work introduces reward teaching for federated multi-armed bandits to guide clients towards global optimality.
problem Existing federated multi-armed bandits designs assume clients will follow the server's protocol, but this is not always feasible.
method Introduces reward teaching where the server adjusts clients' local rewards to encourage global optimality, using phased Teaching-After-Learning (TAL) and Teaching-While-Learning (TWL) algorithms.
result Demonstrates that TAL achieves logarithmic regrets with only logarithmic adjustment costs, and TWL outperforms TAL for UCB1 clients.
Study designs logging policies to minimize off-policy evaluation error.
problem Minimizing OPE error with logging policies for target policies.
method Characterizes reward-coverage tradeoff, proposes a unifying framework, derives optimal policies.
result Provides actionable guidance for firms choosing recommendation systems.
New algorithms improve best-arm identification with varying rewards.
problem Identifying the best arm with varying reward variances in fixed budget.
method Proposed two algorithms: SHVar for known variances, SHAdaVar for unknown variances; uses non-uniform budget allocation.
result Bounding misidentification probabilities for both algorithms.
This paper optimizes driver repositioning using MARL and reward design for better service and traffic management.
problem Unserved passenger requests due to drivers' cruising behavior during passenger seeking.
method Mean field multi-agent reinforcement learning (MARL) with a reward design scheme and Bayesian optimization (BO) to solve bilevel optimization problems.
result Optimal toll charges and service charges can improve platform and city planner objectives by significant margins, leading to better traffic conditions.
New algorithm uses GNNs to optimize rewards in graph-structured data.
problem Optimizing rewards in molecule design with graph-structured data.
method Embedding permutation invariance into GNNs and using GNTK for regret bounds.
result First GNN confidence bound and phased-elimination algorithm with sublinear regret.
Study of repeated games with unobserved agent rewards using MAB framework.
problem Designing policies for principals in repeated principal-agent games with unobservable agent rewards.
method Developed a policy achieving low regret (square-root regret up to a log factor) for perfect-knowledge agents.
result Constructed an estimator for agent's expected reward and designed a policy achieving low regret.
New algorithms for generalized linear bandits with unknown reward functions.
problem Misspecification of reward functions in existing bandit algorithms.
method Introducing single index bandits, proposing STOR, ESTOR, and GSTOR algorithms.
result Achieved nearly optimal regret bound of .
This paper analyzes switchback experiments in A/B testing, revealing key factors affecting their effectiveness.
problem Understanding the effectiveness of different switchback designs in A/B testing.
method Comprehensive comparative analysis of various switchback designs in Markovian environments, covering state-of-the-art RL estimators.
result The effectiveness of switchback designs depends on the size of the carryover effect and reward autocorrelations.
One obstacle to applying reinforcement learning algorithms to real-world problems is the lack of suitable reward functions. Designing such reward functions is difficult in part because the user only has an implicit understanding of the task objective. This gives rise to the agent alignment problem: how do we create age…
SIBRE boosts reinforcement learning convergence by rewarding improvement over past performance.
problem Improving the rate of convergence in reinforcement learning.
method SIBRE is a reward shaping approach that rewards improvement over the agent's own past performance.
result SIBRE converges faster and more stably to the optimal policy compared to baseline RL algorithms.
New algorithm tackles nonstationary linear bandits with latent dynamics.
problem Nonstationary bandit problem with latent states and unknown dynamics.
method Explore-then-commit algorithm with exploration and commitment phases.
result Achieves regret.
Study designs steering rewards for MFGs with unknown dynamics and model uncertainty.
problem Designing incentives for large populations of agents in MFGs with uncertain model details.
method Developed optimistic exploration algorithms for agents with no-adaptive regret behaviors.
result Sub-linear regret guarantees for cumulative gaps between agent behaviors and desired outcomes.
Resolves spurious correlations in causal models via intervention design.
problem Spurious correlations lead to incorrect causal models in reinforcement learning environments.
method Proposes a method to design interventions that improve causal models by incentivizing agents to find errors.
result Experimental results show improved causal models compared to baselines.
f-GRPO and Beyond: Divergence-Based Reinforcement Learning Algorithms for General LLM Alignmentcs.LG
Extends reinforcement learning alignment to scalar rewards, improving math reasoning.
problem Designing reinforcement learning algorithms for general LLM alignment.
method Introduces f-GRPO and f-HAL, estimating f-divergences between reward-aligned and unaligned distributions.
result Improves math reasoning RLVR tasks and mitigates reward hacking.
A new method optimizes in nonstationary environments with many arms efficiently.
problem Optimizing in nonstationary environments with a large number of arms.
method Gaussian interpolation to learn continuous Lipschitz reward functions in nonstationary environments.
result Efficiently learns continuous Lipschitz reward functions with cumulative regret.
New RL approach learns dynamic VCG mechanisms in unknown MDP environments.
problem Learning dynamic VCG mechanisms in unknown MDP environments.
method Reward-free online RL for exploration, combined with function approximation.
result Regret bound of for dynamic VCG mechanism learning.
New method steers protein design towards desired properties.
problem Challenges in designing proteins with specific structures and properties.
method Feynman-Kac framework applied to RFdiffusion models with guiding potentials.
result Significant improvement in predicted interface energetics and binder designability.
A method to train multi-agent reinforcement learning models without intrinsic rewards.
problem Training multi-agent reinforcement learning models with sparse rewards is challenging.
method A learning-based exploration strategy using variational graph autoencoder to generate initial states.
result The method improves the training and performance of multi-agent reinforcement learning models.
Algorithm optimizes system design and control for better rewards.
problem Optimizing system design and control for maximum rewards.
method Deep reinforcement learning combining policy gradient and model-based optimization.
result DEPS algorithm outperforms state-of-the-art methods in various environments.
New algorithm reduces worst-case regret for heavy-tailed bandits.
problem Stochastic Multi-Armed Bandit problem with heavy-tailed rewards.
method Modified minimax policy MOSS with saturated empirical mean.
result Worst-case regret matching lower bound for heavy-tailed distributions.
Deep reinforcement learning achieves superhuman performance in a range of video game environments, but requires that a designer manually specify a reward function. It is often easier to provide demonstrations of a target behavior than to design a reward function describing that behavior. Inverse reinforcement learning …
Recent advances in deep reinforcement learning algorithms have shown great potential and success for solving many challenging real-world problems, including Go game and robotic applications. Usually, these algorithms need a carefully designed reward function to guide training in each time step. However, in real world, …
This paper addresses reward estimation and incentive design for agents with hidden rewards.
problem Estimating and incentivizing agents with unknown rewards in a learning setting.
method Repeated adverse selection game with a self-interested learning agent and a learning principal. Introduces an estimator for consistent reward estimation and a data-driven incentive policy.
result Finite-sample consistency of the estimator and a rigorous regret bound for the principal.
Paper proposes RRD to learn proxy rewards for sparse delayed rewards in episodic reinforcement learning.
problem Learning from sparse and delayed rewards in reinforcement learning.
method Randomized Return Decomposition (RRD) algorithm to redistribute rewards.
result Substantial improvement over baseline algorithms in experiments.
New model-free RL algorithm tackles robust average-reward problems with finite sample complexity analysis.
problem Long-term decision-making in environments with varying dynamics.
method Proposes Robust Halpern Iteration (RHI) algorithm based on a black-box sampling oracle and multi-level Monte-Carlo estimator.
result Achieves ε-optimal robust policy with sample complexity of O(1/ε^(2+o(1))) under generative model setting.
Novel RL approach for molecular design using quantum mechanics.
problem Existing RL methods for molecular design are limited in scope and reward function.
method Formulation in Cartesian coordinates, direct use of quantum mechanics for reward function, translation and rotation invariant state-action space.
result Agent efficiently learns to solve molecular design tasks from scratch.
Reinforcement learning usually makes use of numerical rewards, which have nice properties but also come with drawbacks and difficulties. Using rewards on an ordinal scale (ordinal rewards) is an alternative to numerical rewards that has received more attention in recent years. In this paper, a general approach to adapt…
The study categorizes reward errors in reinforcement learning, finding some can be beneficial.
problem Training language models with imperfect proxy rewards.
method Theoretical analysis of policy gradient optimization and categorization of reward errors.
result Reward errors can be benign or even beneficial, preventing policy from stalling.
Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decodingcs.LG
New method optimizes diffusion models without fine-tuning, integrating soft value functions.
problem Optimizing natural design spaces of images, molecules, DNA, RNA, and protein sequences.
method Iterative sampling method integrating soft value functions into diffusion model inference.
result Directly utilizes non-differentiable features/reward feedback, applies to discrete diffusion models.
Paper tackles robust batched bandits for heavy-tailed rewards.
problem Clinical trials and other applications with heavy-tailed rewards.
method Proposes robust batched bandit algorithms for heavy-tailed rewards in finite-arm and Lipschitz-continuous settings.
result Heavier-tailed rewards require fewer batches for near-optimal regret in the instance-independent regime and Lipschitz setting.
New RL approach infers optimal policies via variational inference.
problem Manual design of reward functions for reinforcement learning.
method Variational inference for inferring policies achieving desired outcomes.
result Eliminates need for hand-crafted reward functions for diverse tasks.