In this paper, we propose to combine imitation and reinforcement learning via the idea of reward shaping using an oracle. We study the effectiveness of the near-optimal cost-to-go oracle on the planning horizon and demonstrate that the cost-to-go oracle shortens the learner's planning horizon as function of its accurac…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “planning horizons”
In reinforcement learning, the discount factor controls the agent's effective planning horizon. Traditionally, this parameter was considered part of the MDP; however, as deep reinforcement learning algorithms tend to become unstable when the effective planning horizon is long, recent works refer to as a hyper-p…
This work tackles long-term visual planning by goal-conditioned hierarchical predictors.
problem Current learning approaches fail on long-horizon tasks due to lack of goal information and coarse-to-fine planning.
method Formulate goal-conditioned predictors (GCPs) and hierarchical models to predict trajectories between observations.
result GCPs enable effective long-term planning with much longer horizons than before.
Despite its potential to improve sample complexity versus model-free approaches, model-based reinforcement learning can fail catastrophically if the model is inaccurate. An algorithm should ideally be able to trust an imperfect model over a reasonably long planning horizon, and only rely on model-free updates when the …
Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcs.LG
Video prediction models combined with planning algorithms have shown promise in enabling robots to learn to perform many vision-based tasks through only self-supervision, reaching novel goals in cluttered scenes with unseen objects. However, due to the compounding uncertainty in long horizon video prediction and poor s…
This work defines a complexity measure for BAMDP planning and introduces state abstraction for more efficient approximate planning.
problem The computational intractability of exact BAMDP planning solutions.
method Define a complexity measure for BAMDP planning, introduce state abstraction, and develop an approximate planning algorithm.
result Introduces a computationally tractable approximate planning algorithm using state abstraction.
New algorithm for RL with horizon-free reward-free exploration for linear MDPs.
problem Reward-free reinforcement learning with long planning horizons.
method Uncertainty-weighted value-targeted regression with exploration-driven pseudo-reward and moment estimator.
result Horizon-free sample complexity of for finding an -optimal policy.
Is Long Horizon Reinforcement Learning More Difficult Than Short Horizon Reinforcement Learning?cs.LG
Long horizon reinforcement learning is as hard as short horizon learning.
problem Understanding the difficulty of long horizon reinforcement learning problems.
method Introduced new concepts: ε-net for optimal policies and Online Trajectory Synthesis algorithm.
result Proved that sample complexity scales logarithmically with the planning horizon, refuting the conjecture.
Agents compose pre-trained policies for complex tasks, improving zero-shot performance.
problem Challenges in long-horizon predictions and estimating visitation distributions induced by policy sequences.
method Learn predictive jumpy world models of multi-step dynamics, enhancing predictions with a consistency objective.
result Compositional planning with jumpy world models yields, on average, a 200% relative improvement over primitive actions on long-horizon tasks.
The objective of this work is to augment the basic abilities of a robot by learning to use new sensorimotor primitives to enable the solution of complex long-horizon problems. Solving long-horizon problems in complex domains requires flexible generative planning that can combine primitive abilities in novel combination…
SGM combines deep learning and planning for robust long-horizon tasks.
problem Combining deep learning and planning for robust long-horizon tasks.
method Sparse Graphical Memory (SGM) that stores states and feasible transitions in a sparse memory, aggregating states according to a two-way consistency objective.
result SGM significantly outperforms current state of the art methods on long horizon, sparse-reward visual navigation tasks.
UCRL-WVTR tackles long-term reinforcement learning with general approximations, achieving horizon-free and instance-dependent regret bounds.
problem Long-term reinforcement learning with general function approximations.
method UCRL-WVTR proposes a novel algorithm, UCRL-WVTR, with weighted value-targeted regression and a high-order moment estimator.
result Achieves horizon-free and instance-dependent regret bounds matching minimax lower bounds up to logarithmic factors.
New research shows exponential lower bounds for planning in MDPs with linearly-realizable optimal action-value functions.
problem Determining the minimum number of queries needed for sound planners in MDPs with linear function approximation.
method Analyzing fixed-horizon and discounted MDPs with a generative model, showing lower bounds on the number of queries required.
result Sound planners need at least exponential number of queries in both fixed-horizon and discounted settings.
We present HiDe, a novel hierarchical reinforcement learning architecture that successfully solves long horizon control tasks and generalizes to unseen test scenarios. Functional decomposition between planning and low-level control is achieved by explicitly separating the state-action spaces across the hierarchy, which…
In this paper, we propose a novel Reinforcement Learning approach for solving the Active Information Acquisition problem, which requires an agent to choose a sequence of actions in order to acquire information about a process of interest using on-board sensors. The classic challenges in the information acquisition prob…
C-Learning estimates reachability over time to solve multi-goal tasks.
problem Multi-goal reaching challenges in reinforcement learning.
method Cumulative accessibility functions and recurrence relations.
result Optimal cumulative accessibility functions are monotonic in horizon.
New algorithm reduces reinforcement learning complexity, approaching contextual bandits.
problem Episodic reinforcement learning's difficulty compared to contextual bandits.
method Proposes MVP algorithm with a new Bernstein-type bonus for episodic reinforcement learning.
result Achieves near-optimal regret bound of $O\left(\left(\sqrt{SAK} + S^2A
ight) \poly\log \left(SAHK
ight)
ight)$, improving state-of-the-art results.
Paper analyzes neural network complexity for planning problems.
problem Understanding neural network complexity for planning policies.
method Circuit complexity analysis for relational neural networks.
result Three classes of planning problems identified based on network complexity.
TD-Flow improves long-term predictions in agent learning.
problem Cumulative errors in step-by-step inference of future states.
method Leverages flow-matching techniques and a novel Bellman equation to learn accurate geometric horizon models.
result Significantly reduces errors at long horizons compared to prior methods.
State-of-the-art efficient model-based Reinforcement Learning (RL) algorithms typically act by iteratively solving empirical models, i.e., by performing \emph{full-planning} on Markov Decision Processes (MDPs) built by the gathered experience. In this paper, we focus on model-based RL in the finite-state finite-horizon…
On Query-efficient Planning in MDPs under Linear Realizability of the Optimal State-value Functioncs.LG
TensorPlan algorithm finds δ-optimal policies with poly queries under linearly realizable state-value function.
problem Efficient planning in MDPs with linearly realizable state-value function.
method TensorPlan algorithm using poly simulator queries.
result First algorithm with polynomial query complexity using only linear-realizability of a single competing value function.
New method for optimistic planning in MDPs using regularization.
problem Optimistic planning in infinite-horizon discounted MDPs.
method Regularized dynamic programming for approximate value iteration.
result Achieves near-optimal statistical guarantees in learning policies.
Efficiently plans large MDPs with weak function approximations.
problem Planning in large MDPs with limited function approximation capabilities.
method Uses linear value function approximation with weak requirements and a generative oracle.
result Produces almost-optimal actions for any state with polynomial computation time.
Random investment strategies outperform sensible ones, even with forecasts.
problem The usefulness of investment strategies based on forecasts is questioned.
method Investigated the performance of sensible and nonsensical investment strategies, including forecasts.
result There is no substantial difference between the performances of ``best'' and ``trivial'' forecasts.
Plan2Vec learns image representations without labels, improving control tasks.
problem Learning image representations without labeled data.
method Constructs a weighted graph using near-neighbor distances and extrapolates to global embedding.
result Plan2Vec achieves accurate long-term value estimates in control tasks with reduced computational and memory costs.
Real Time Dynamic Programming (RTDP) is an online algorithm based on Dynamic Programming (DP) that acts by 1-step greedy planning. Unlike DP, RTDP does not require access to the entire state space, i.e., it explicitly handles the exploration. This fact makes RTDP particularly appealing when the state space is large and…
The hierarchical structure of production planning has the advantage of assigning different decision variables to their respective time horizons and therefore ensures their manageability. However, the restrictive structure of this top-down approach implying that upper level decisions are the constraints for lower level …
The paper studies risk-sensitive MDPs with recursive risk measures.
problem Risk-sensitive decision-making in MDPs with unbounded costs.
method Recursive application of static risk measures, Bellman equation derivation, existence of optimal policies.
result Existence of Markovian optimal policies for infinite planning horizons, contractive model for stationary optimal policy.
Olympic Games consistently exceed budgets, leading to unpredictable costs.
problem High costs and unpredictability of the Olympic Games.
method Statistical analysis of historical data to explain cost risks.
result Olympic costs follow a power-law distribution with infinite mean and variance.
Optimal timing for borrowing from a 457(b) plan to maximize returns.
problem Deciding the best time to borrow from a tax-advantaged retirement account.
method Formulated and solved the optimal stopping problem for a loan from a 457(b) plan.
result Derived cutoff rules for optimal loan control, showing how to wait until a certain amount of money is accumulated.
We solve non-Markovian optimal switching problems in discrete time on an infinite horizon, when the decision maker is risk aware and the filtration is general, and establish existence and uniqueness of solutions for the associated reflected backward stochastic difference equations. An example application to hydropower …
When the planning horizon is long, and the safe asset grows indefinitely, isoelastic portfolios are nearly optimal for investors who are close to isoelastic for high wealth, and not too risk averse for low wealth. We prove this result in a general arbitrage-free, frictionless, semimartingale model. As a consequence, op…
Model learns and plans in real-time under constraints for robotic systems.
problem Challenges in applying reinforcement learning to robotic systems, especially handling continuous state and action spaces, time/resource budget, and hard constraints.
method Combines Gaussian Process regression and Receding Horizon Control. Uses sparse spectrum Gaussian Processes for incremental model updates from sensory data.
result Demonstrates benefits of online learning on autonomous racing tasks and reusability of learned dynamics.
MPC outperforms reactive budgeting in non-stationary return environments.
problem Optimizing budget allocation under non-stationary returns.
method Receding-horizon Model Predictive Control (MPC) compared to reactive policies.
result MPC consistently outperforms reactive budgeting when return dynamics are predictable.
TensorPlan shows an exponential lower bound for planning in MDPs with linearly realizable value functions.
problem Finding an exponential lower bound for planning in MDPs with linearly realizable value functions.
method TensorPlan and a few action lower bound approach.
result An exponentially large lower bound is shown for planning in MDPs with linearly realizable value functions.
RL policy tracks dynamic targets in partially known environments robustly.
problem Active target tracking in partially known environments.
method Deep reinforcement learning (RL) approach for in-sight tracking, navigation, and exploration.
result Unified RL policy shows robust behavior for agile and anomalous targets.
We propose a plan online and learn offline (POLO) framework for the setting where an agent, with an internal model, needs to continually act and learn in the world. Our work builds on the synergistic relationship between local model-based control, global value function learning, and exploration. We study how local traj…
We consider the portfolio choice problem for a long-run investor in a general continuous semimartingale model. We suggest to use path-wise growth optimality as the decision criterion and encode preferences through restrictions on the class of admissible wealth processes. Specifically, the investor is only interested in…
New RL method learns K-step lookahead Q-functions for fixed-horizon MDPs.
problem Challenges in online reinforcement learning for non-episodic, finite-horizon MDPs.
method Introduces a K-step lookahead Q-function with a time-varying threshold for selecting actions.
result Achieves minimax optimal constant regret for K=1 and regret for K ≥ 2.
Study forecasts sub-city real estate prices weekly using radar and news sentiment.
problem Limited availability of reliable real estate price indicators at neighborhood and long horizons.
method Combining satellite radar signals and news sentiment to forecast sub-city real estate prices.
result The multimodal model reduces mean absolute error by 35% at long horizons (26-34 weeks).
Multi-period Mean-Buffered Probability of Exceedance in Defined Contribution Portfolio Optimizationq-fin.PM
Investigates multi-period portfolio optimization for DC plans using buffered Probability of Exceedance.
problem Optimizing long-term Defined Contribution plans with realistic constraints and dynamic dynamics.
method Formulates and solves bilevel optimization problems for pre-commitment and time-consistent Mean-bPoE and Mean-CVaR portfolio optimization.
result Time-consistent Mean-bPoE strategies maintain investor preferences for minimum terminal wealth, unlike Mean-CVaR.
Exploration in reinforcement learning (RL) suffers from the curse of dimensionality when the state-action space is large. A common practice is to parameterize the high-dimensional value and policy functions using given features. However existing methods either have no theoretical guarantee or suffer a regret that is ex…
In many finite horizon episodic reinforcement learning (RL) settings, it is desirable to optimize for the undiscounted return - in settings like Atari, for instance, the goal is to collect the most points while staying alive in the long run. Yet, it may be difficult (or even intractable) mathematically to learn with th…
Efficient local planning with linear approximations for agents with limited simulator access.
problem Planning with limited simulator access in reinforcement learning.
method Confident Monte Carlo Least Square Policy Iteration (Confident MC-LSPI) and Politex (Confident MC-Politex) algorithms.
result The algorithms can learn the optimal policy with local simulator access, even for linear Q-functions.
We propose a framework for general probabilistic multi-step time series regression. Specifically, we exploit the expressiveness and temporal nature of Sequence-to-Sequence Neural Networks (e.g. recurrent and convolutional structures), the nonparametric nature of Quantile Regression and the efficiency of Direct Multi-Ho…
This paper studies a recent proposal to use randomized value functions to drive exploration in reinforcement learning. These randomized value functions are generated by injecting random noise into the training data, making the approach compatible with many popular methods for estimating parameterized value functions. B…
In this short note we consider a dynamic assortment planning problem under the capacitated multinomial logit (MNL) bandit model. We prove a tight lower bound on the accumulated regret that matches existing regret upper bounds for all parameters (time horizon , number of items and maximum assortment capacity )…
Algorithm learns multiple tasks with minimal planning, achieving near-optimal performance.
problem Learning multiple tasks efficiently in a reinforcement learning setting.
method UCB Lifelong Value Distillation (UCBlvd) algorithm with structural assumption for shared exploration.
result Sublinear regret bound of with planning calls.