Reward tweaking optimizes behavior for long-term goals by adjusting the reward function.
problem Optimizing behavior for long-term goals in reinforcement learning with unstable long planning horizons.
method Reward tweaking learns a surrogate reward function that induces optimal behavior for the original task.
result Reward tweaking guides agents towards better long-term returns while planning for short horizons.
New algorithms for efficient learning with long-term rewards in contextual bandits.
problem Efficient learning with long-term rewards in contextual bandits.
method Proposes new algorithms leveraging sparsity to discover dependence patterns and arm parameters.
result Regret upper bounds for data-poor and data-rich regimes, showing improved sample complexity.
Framework uses expert intervention to solve long-horizon reinforcement learning tasks.
problem Long horizon robot learning tasks with sparse rewards.
method Option templates and expert intervention to enable high-level task understanding.
result Framework outperforms state-of-the-art approaches by two orders of magnitude.
New algorithm for RL with horizon-free reward-free exploration for linear MDPs.
problem Reward-free reinforcement learning with long planning horizons.
method Uncertainty-weighted value-targeted regression with exploration-driven pseudo-reward and moment estimator.
result Horizon-free sample complexity of O ( d 2 ε − 2 ) O(d^2\varepsilon^{-2}) O ( d 2 ε − 2 ) for finding an ε \varepsilon ε -optimal policy. Learning to imitate expert behavior from demonstrations can be challenging, especially in environments with high-dimensional, continuous observations and unknown dynamics. Supervised learning methods based on behavioral cloning (BC) suffer from distribution shift: because the agent greedily imitates demonstrated action…
Long horizon reinforcement learning is as hard as short horizon learning.
problem Understanding the difficulty of long horizon reinforcement learning problems.
method Introduced new concepts: ε-net for optimal policies and Online Trajectory Synthesis algorithm.
result Proved that sample complexity scales logarithmically with the planning horizon, refuting the conjecture.
Paper proposes RRD to learn proxy rewards for sparse delayed rewards in episodic reinforcement learning.
problem Learning from sparse and delayed rewards in reinforcement learning.
method Randomized Return Decomposition (RRD) algorithm to redistribute rewards.
result Substantial improvement over baseline algorithms in experiments.
Algorithm reduces long-term policy regret in ML decision-making.
problem Capturing long-term impacts of ML decisions in communities.
method Modeling communities as arms in a multi-armed bandit problem, defining policy regret as a stronger metric than external regret.
result Algorithm achieves provably sub-linear policy regret for long time horizons.
New algorithm tackles nonstationary linear bandits with latent dynamics.
problem Nonstationary bandit problem with latent states and unknown dynamics.
method Explore-then-commit algorithm with exploration and commitment phases.
result Achieves i l d e O ( T 2 / 3 ) ilde{\mathcal{O}}(T^{2/3}) i l d e O ( T 2/3 ) regret. SGM combines deep learning and planning for robust long-horizon tasks.
problem Combining deep learning and planning for robust long-horizon tasks.
method Sparse Graphical Memory (SGM) that stores states and feasible transitions in a sparse memory, aggregating states according to a two-way consistency objective.
result SGM significantly outperforms current state of the art methods on long horizon, sparse-reward visual navigation tasks.
Proposes a method to boost deep reinforcement learning with sparse rewards.
problem Challenges in learning complex behaviors with long horizons and sparse rewards.
method Predictive coding for reward shaping.
result Achieves better learning by providing reward signals that understand environment dynamics and emphasize useful features.
We study how to effectively leverage expert feedback to learn sequential decision-making policies. We focus on problems with sparse rewards and long time horizons, which typically pose significant challenges in reinforcement learning. We propose an algorithmic framework, called hierarchical guidance, that leverages the…
This work improves RL for complex robotic tasks by guiding exploration with task-specific goal distributions.
problem Solving long-horizon, complex sequential tasks in robotics with sparse rewards.
method Extends hindsight relabelling to task-specific goal distributions using a small set of demonstrations.
result Significantly higher overall performance on complex robotic manipulation tasks.
The paper tackles batch policy learning in Markov Decision Processes, focusing on average reward maximization.
problem Maximizing long-term average reward in Markov Decision Processes with batch learning.
method Doubly robust estimator for average reward, optimization algorithm for optimal policy, finite-sample regret guarantee.
result The proposed method achieves semiparametric efficiency and provides a finite-sample regret guarantee.
Q-chunking improves RL for long tasks by chunking actions.
problem Improving sample efficiency and exploration in offline-to-online RL.
method Action chunking in TD-based RL methods.
result Q-chunking outperforms prior methods on long-horizon tasks.
Action-bisimulation learns long-horizon controllability for reinforcement learning.
problem Learning relevant state features in high-dimensional observations for robust reinforcement learning.
method Action-bisimulation encoding, inspired by bisimulation invariance, extends single-step controllability to multi-step.
result Action-bisimulation pretraining improves sample efficiency in various environments.
Investors can achieve optimal risk-reward trade-offs with bonds and stocks under mean-reverting stock returns.
problem Optimizing investment strategies with mean-reverting stock returns.
method Calculus of variations to derive the entire family of extremal strategies, not just the optimal ones.
result The value of the portfolio is effectively bounded from below, providing a 'guarantee' on the horizon.
Two new algorithms improve model-free RL for infinite-horizon MDPs.
problem Learning in infinite-horizon average-reward MDPs.
method Two model-free algorithms for infinite-horizon average-reward MDPs.
result Improved regret bounds for model-free RL in MDPs.
Behavior cloning training instabilities amplified by SGD noise over long horizons.
problem Training instabilities in behavior cloning with deep neural networks.
method Empirical dissection of minibatch SGD updates and their effects on long-horizon rewards.
result Exponential moving average (EMA) of iterates effectively mitigates gradient variance amplification (GVA).
We consider the off-policy estimation problem of estimating the expected reward of a target policy using samples collected by a different behavior policy. Importance sampling (IS) has been a key technique to derive (nearly) unbiased estimators, but is known to suffer from an excessively high variance in long-horizon pr…
Despite significant advances in the field of deep Reinforcement Learning (RL), today's algorithms still fail to learn human-level policies consistently over a set of diverse tasks such as Atari 2600 games. We identify three key challenges that any algorithm needs to master in order to perform well on all games: process…
Optimal sample complexity analysis for plug-in approach in average-reward MDPs.
problem Learning optimal policies in average-reward MDPs with a generative model.
method Plug-in approach that constructs a model estimate and computes an optimal policy.
result Optimal sample complexities for the plug-in approach without prior knowledge of problem parameters.
PRISM integrates diverse rewards in MORL, improving sample efficiency and Pareto coverage.
problem Heterogeneous MORL where dense objectives dominate, leading to poor sample efficiency.
method PRISM uses reflectional symmetry and ReSymNet to reconcile temporal-frequency mismatches and accelerate exploration.
result PRISM consistently outperforms sparse-reward baselines and oracles, achieving significant Pareto gains.
Study analyzes Nifty 50 returns over 34 years, showing P/E ratio predicts long-term gains.
problem Understanding equity return dynamics in the Indian market over various horizons.
method Unified, distribution-aware, complexity-informed framework using 34 years of Nifty 50 data.
result P/E ratio probabilistically maps return distributions across different investment horizons.
Reinforcement learning aids decision-making in economics and finance.
problem Optimal decision-making in dynamic, uncertain environments.
method Reinforcement learning algorithms to learn optimal policies.
result Deep learning enhances solving complex behavioral problems.
The study reveals distinct patterns in retail investors' holding periods affecting stock returns.
problem Understanding the impact of retail investors' investment horizons on stock returns.
method Using self-reported holding periods from StockTwits, the study categorizes retail investors into long-horizon and short-horizon groups and analyzes their return patterns.
result Long-horizon retail investors exhibit underreaction to earnings announcements, while short-horizon investors show overreaction.
State-of-the-art forecasting methods using Recurrent Neural Net- works (RNN) based on Long-Short Term Memory (LSTM) cells have shown exceptional performance targeting short-horizon forecasts, e.g given a set of predictor features, forecast a target value for the next few time steps in the future. However, in many appli…
New algorithm reduces reinforcement learning regret to sqrt(T) without strong dynamics assumptions.
problem Infinite-horizon average-reward reinforcement learning with linear MDPs.
method Approximate by discounted-reward MDPs and apply optimistic value iteration.
result Achieves O(sqrt(T)) regret with polynomial complexity.
UCRL2-VTR achieves nearly optimal regret for learning MDPs with linear function approximation.
problem Learning infinite-horizon average-reward MDPs with linear function approximation.
method UCRL2-VTR algorithm with Bernstein-type bonus.
result Achieves a regret of i l d e O ( d D T ) ilde{O}(d\sqrt{DT}) i l d e O ( d D T ) with matching lower bound. This paper balances short-term and long-term rewards in policy learning.
problem Balancing short-term and long-term rewards in policy learning.
method Formalizes a new framework to balance rewards, identifies rewards under mild assumptions, deduces efficiency bounds, and develops a policy learning approach.
result The proposed method improves the estimator of long-term reward and reduces regret.
Method solves long-horizon robotic tasks via imitation and reinforcement learning.
problem Long-horizon robotic tasks with complex sequences of actions.
method Two-phase approach: imitation learning followed by reinforcement learning.
result Method can scale to challenging long-horizon tasks.
Efficient RL for linear MDPs with unknown transitions.
problem Long planning horizons and unknown state transitions in linear mixture MDPs.
method Horizon-free algorithm using weighted least squares with variance and uncertainty awareness.
result Achieves optimal regret up to logarithmic factors.
Study proposes adaptive RL for dynamic portfolio optimization.
problem Traditional portfolio optimization models fail to adapt to regime shifts.
method Regime-aware reinforcement learning framework with hybrid observations and constrained reward functions.
result Transformer PPO achieves highest risk-adjusted returns, while LSTM variants offer a good balance.
A new model-free algorithm achieves near-optimal regret for infinite-horizon MDPs.
problem Model-free reinforcement learning for infinite-horizon average-reward MDPs.
method Exploration Enhanced Q-learning (EE-QL) for weakly communicating MDPs.
result Achieves O ( T ) O(\sqrt{T}) O ( T ) regret bound for general weakly communicating MDPs. New algorithm reduces reinforcement learning complexity, approaching contextual bandits.
problem Episodic reinforcement learning's difficulty compared to contextual bandits.
method Proposes MVP algorithm with a new Bernstein-type bonus for episodic reinforcement learning.
result Achieves near-optimal regret bound of $O\left(\left(\sqrt{SAK} + S^2A
ight) \poly\log \left(SAHK
ight)
ight)$ , improving state-of-the-art results.
This work improves Q-learning for average-reward MDPs, reducing sample and communication complexities in federated settings.
problem Improving sample complexity of Q-learning for average-reward MDPs.
method Simple Q-learning algorithm with carefully chosen parameters for both single-agent and federated scenarios.
result Established first federated Q-learning algorithm for average-reward MDPs with provable efficiency in sample and communication complexities.
New method for efficient online exploration in RLHF reduces regret.
problem Efficiently collecting new preference data in RLHF to refine reward model and policy.
method Proposes a new exploration scheme that directs preference queries toward reducing uncertainty in reward differences most relevant to policy improvement.
result Establishes regret bounds of order T ( β + 1 ) / ( β + 2 ) T^{(β+1)/(β+2)} T ( β + 1 ) / ( β + 2 ) for online RLHF, with polynomial scaling in all model parameters. The paper tackles non-stationary MAB with periodic rewards.
problem Non-stationary mean rewards over time in a business context.
method Combines Fourier analysis with confidence-bound learning to estimate periods and minimize regret.
result Proposes a near-optimal policy with a regret bound of O ( T ∑ k = 1 K T k ) O(\sqrt{T\sum_{k=1}^K T_k}) O ( T ∑ k = 1 K T k ) . Study long-term asset liquidation behavior with external flows.
problem Investigate optimal liquidation in presence of external flows.
method Convergence analysis of BSDEs for value function and strategy.
result Long-term liquidation may not occur due to external flows.
New algorithm optimizes for long-term user satisfaction in delayed reward settings.
problem Optimizing for long-term user satisfaction in delayed reward settings.
method Developed a predictive model of delayed rewards and a bandit algorithm that combines rewards and surrogate outcomes.
result Our algorithm significantly outperforms methods that optimize for short-term proxies or rely solely on delayed rewards.
Reservoir computing predicts chaotic systems for long horizons with sparse updates.
problem Predicting chaotic systems with long horizons using limited data.
method Sparse, time-dependent data inputs into reservoir computing.
result Achieves arbitrarily long prediction horizons for chaotic systems.
In this paper, we propose to combine imitation and reinforcement learning via the idea of reward shaping using an oracle. We study the effectiveness of the near-optimal cost-to-go oracle on the planning horizon and demonstrate that the cost-to-go oracle shortens the learner's planning horizon as function of its accurac…
Paper optimizes recommendation systems for long-term business metrics.
problem Short-term reward optimization ignores long-term business metrics.
method Introduced a framework for modeling long-term rewards in RecoGym.
result Proposed a simple extension leading to state-of-the-art results.
A policy for near-optimal multi-player bandits with non-zero collision rewards.
problem Decentralized multi-player bandits with heterogeneous rewards and collisions.
method A policy achieving near-optimal regret in a non-communicative setting.
result Near order-optimal expected regret of O ( log 1 + δ T ) O(\log^{1 + δ} T) O ( log 1 + δ T ) for 0 < δ < 1 0 < δ< 1 0 < δ < 1 . The paper studies reward concentration in MDPs, covering asymptotic and non-asymptotic settings.
problem Reward concentration in Markov Decision Processes (MDPs).
method Unified approach to reward concentration in MDPs, including asymptotic and non-asymptotic bounds.
result Rate-equivalent definitions of regret for learning policies.
DMIDAS improves long-term forecasting accuracy in healthcare and electricity data.
problem Challenging long-term forecasting accuracy and computational complexity.
method Smoothness regularization and mixed data sampling techniques integrated into NBEATS architecture.
result Improves prediction accuracy by 5% on long forecasting horizons (1000 timestamps) compared to state-of-the-art models.
Optimizes Ethena's yield strategy by controlling stETH and ETH futures positions.
problem Maximizes Ethena's yield while managing price impacts.
method Formulates and solves stochastic control problems for Ethena's yield-generating strategy.
result Explicitly determines optimal control rates for stETH and ETH futures.
RLVR training dynamics reveal an implicit curriculum that shapes learning progression.
problem Understanding how RLVR overcomes the long-horizon barrier.
method Developed a theory of training dynamics for RLVR on transformers, using Fourier analysis on finite groups.
result Mixed-difficulty training naturally follows an implicit curriculum, shaping the learning progression from easy to hard.