New RL method reduces sample complexity for large policy spaces.
problem Large-scale RL with unknown optimal policies and state/action spaces.
method Introduces eluder dimension for policy space, proving near-optimal sample complexity.
result Near-optimal sample complexity upper bound that depends linearly on eluder dimension.
New method reduces bias in high-dimensional action spaces for efficient reinforcement learning.
problem Large bias and difficulty in reusing old samples in high-dimensional action spaces.
method Dimension-wise IS weight clipping to control bias and adaptively manage IS weights.
result Proposed method outperforms PPO and other RL algorithms in various tasks.
Policy analysts wish to visualize a range of policies for large simulator-defined Markov Decision Processes (MDPs). One visualization approach is to invoke the simulator to generate on-policy trajectories and then visualize those trajectories. When the simulator is expensive, this is not practical, and some method is r…
Study shows PI for LQR requires fewer policy improvement steps than policy evaluation steps.
problem Understanding the sample complexity of RL algorithms for continuous control tasks.
method Finite-time analysis of approximate policy iteration for LQR, quantifying policy improvement and evaluation complexities.
result Policy evaluation is the dominant factor in sample complexity, requiring ( n + d ) 3 / ε 2 (n+d)^3/\varepsilon^2 ( n + d ) 3 / ε 2 samples per step. It has long been assumed that high dimensional continuous control problems cannot be solved effectively by discretizing individual dimensions of the action space due to the exponentially large number of bins over which policies would have to be learned. In this paper, we draw inspiration from the recent success of sequ…
AdaRL improves robust RL by adaptively adjusting policy complexity.
problem Handling epistemic uncertainty in environment dynamics.
method Bi-level optimization framework with adaptive rank adjustment.
result AdaRL outperforms existing methods on MuJoCo benchmarks.
Paper combines RL with policy regularization for inventory policies.
problem Optimizing inventory policies using RL and dynamic programming.
method Hybrid approach combining RL with policy regularization.
result Generalization guarantees for inventory policies using VC theory.
Study optimal policy regret in partially observable Markov games with adaptive opponents.
problem Optimal sequential decision-making in partially observable environments against strategic, adaptive opponents.
method An epoch-based optimistic maximum-likelihood algorithm that selects one policy per epoch using confidence sets built cumulatively from past data.
result Achieves i l d e O ( T ) ilde{O}(\sqrt{T}) i l d e O ( T ) policy regret for fixed problem parameters, with explicit dependence on horizon, adversary memory, confidence radius, and aggregate Eluder dimension. ENIAC method optimizes and explores complex RL problems with non-linear policies.
problem Theoretical understanding of non-linear policies in RL with strategic exploration.
method ENIAC, an actor-critic method for non-linear function approximation.
result ENIAC finds near-optimal policies in polynomial exploration rounds under bounded eluder dimension.
New method learns policies without limiting to Gaussian distributions.
problem Limitations of Gaussian parameterization in policy learning.
method Advantage weighted quantile regression for implicit policy modeling.
result Comparable or superior performance on MuJoCo benchmarks.
Enhances PMD with lookahead to improve RL performance.
problem Improving RL performance with greedy policies over 1-step.
method Integrates multi-step greedy policies into PMD with lookahead.
result Shows faster convergence rate for h h h -PMD. The paper proposes autoregressive models for better offline RL.
problem Offline RL policy evaluation and optimization challenges.
method Autoregressive dynamics models for sequential state and reward prediction.
result Autoregressive models outperform standard methods in log-likelihood and RL tasks.
New RL algorithm tackles complex discrete action spaces.
problem Challenges in applying on-policy RL in high-dimensional discrete action spaces.
method Action-value critic, correlated actions, gradient sparsification.
result Empirically outperforms related on-policy algorithms.
New algorithm reduces complexity in multi-agent reinforcement learning.
problem High computational complexity in exact computations for multi-agent reinforcement learning.
method Design of a scalable algorithm based on Natural Policy Gradient, using local information and limited communication.
result Converges to globally optimal policy with dimension-free complexity and localization error.
Paper improves policy updates in reinforcement learning to speed up learning.
problem Slow learning and unlearning in policy optimization.
method Introduces a novel gradient update and a modified policy update.
result Proves modified policy update converges to global optimality.
OPPO optimizes policy learning with provable efficiency.
problem Designing efficient policy optimization algorithms that incorporate exploration.
method Proposes an Optimistic Proximal Policy Optimization (OPPO) algorithm.
result Achieves i l d e O ( d 2 H 3 T ) ilde{O}(\sqrt{d^2 H^3 T} ) i l d e O ( d 2 H 3 T ) regret in episodic MDPs. This work explores representation complexity in RL paradigms, revealing model-based RL as the easiest task.
problem Investigating the representation complexity gap among model-based, policy-based, and value-based RL.
method Demonstrated through analysis of Markov decision processes (MDPs) and introduced new classes of MDPs.
result Representation complexity hierarchy: model-based RL > policy-based RL > value-based RL.
Optimistic NPG improves policy optimization in online RL with efficient sample complexity.
problem Limited theoretical understanding of policy optimization, especially in online RL.
method Combines natural policy gradient with optimistic policy evaluation.
result Achieves optimal dimension dependence sample complexity for learning near-optimal policies.
The paper analyzes the sample complexities for policy evaluation with linear function approximation.
problem Policy evaluation with linear function approximation in discounted infinite horizon Markov decision processes.
method Investigates sample complexities for two policy evaluation algorithms: TD and TDC.
result Establishes high-probability sample complexity bounds for policy evaluation algorithms.
RPG improves sample efficiency in RL by learning optimal action ranks.
problem Sample inefficiency in reinforcement learning.
method Ranking Policy Gradient (RPG) method that learns optimal rank of discrete actions.
result RPG reduces sample complexity and improves sample efficiency for large-scale problems.
Policy evaluation is a crucial step in many reinforcement-learning procedures, which estimates a value function that predicts states' long-term value under a given policy. In this paper, we focus on policy evaluation with linear function approximation over a fixed dataset. We first transform the empirical policy evalua…
Study shows high-dimensional sparse RL hardness and Lasso Q-iteration's nearly dimension-free regret.
problem Hardness of online sparse reinforcement learning in high-dimensional MDPs.
method Lower bound construction and Lasso fitted Q-iteration analysis.
result Lasso Q-iteration achieves nearly dimension-free regret of O ~ ( s 2 / 3 N 2 / 3 ) \tilde{O}(s^{2/3}N^{2/3}) O ~ ( s 2/3 N 2/3 ) with oracle access to a good exploratory policy. GPE algorithm optimizes nonparametric contextual bandits with efficient regret bounds.
problem Optimizing nonparametric contextual bandits with efficient regret bounds.
method Inspired by Policy Elimination, GPE uses oracle-efficient techniques for nonparametric classes with infinite VC-dimension.
result GPE is regret-optimal for policy classes with integrable entropy, and for larger entropy, it provides an ε \varepsilon ε -greedy algorithm with matching regret bounds. New framework optimizes multi-asset portfolio choice for high dimensions.
problem Optimizing high-dimensional continuous-time portfolio choice.
method Combines Pontryagin's Maximum Principle with BPTT for neural network policy learning.
result Achieves near-optimal policies with improved efficiency and precision.
Estimates policy performance in small-data settings without sacrificing data.
problem Poor performance of cross-validation in small-data optimization.
method Uses sensitivity analysis to estimate gradient of optimal objective value.
result Explicit high-probability bounds on error of estimator for small-data, large-scale problems.
The paper provides a non-asymptotic error bound for linear system identification under nonlinear policies.
problem System identification for linear systems with nonlinear and/or time-varying policies under i.i.d. random excitation noises.
method Least square estimation with non-asymptotic error bound for bounded state and action trajectories.
result The error bound is consistent with linear policies and generalizes existing guarantees.
Study derivative-free methods for linear policies in linear-quadratic systems.
problem Optimizing policies in linear-quadratic systems with limited derivative information.
method Derivative-free methods applied to linear policies over various noise and reward feedback settings.
result These methods converge to near-optimal policies with a polynomial number of zero-order evaluations.
New algorithm for model selection in contextual bandits reduces regret.
problem Adapting to the complexity of the optimal policy in contextual bandits.
method Designing an algorithm that balances exploration and exploitation, achieving optimal regret bounds.
result Achieves i l d e O ( T 2 / 3 d m ⋆ 1 / 3 ) ilde{O}(T^{2/3}d^{1/3}_{m^\star}) i l d e O ( T 2/3 d m ⋆ 1/3 ) regret with no prior knowledge of the optimal dimension d m ⋆ d_{m^\star} d m ⋆ . GPMD solves regularized RL with linear convergence, promoting structural policies.
problem Regularized reinforcement learning to encourage exploration and structural policies.
method Policy mirror descent with generalized convex regularizers and Bregman divergence.
result GPMD converges linearly to the global solution over a wide range of learning rates.
Estimates and infers multi-stage stationary treatment policies with variable selection.
problem Valid inference for multi-stage stationary treatment policies with high-dimensional feature variables.
method Estimate the value function using augmented inverse probability weighted estimator, apply penalty for variable selection, construct one-step improvements for valid inference.
result Improved estimators are asymptotically normal, valid inference for policy parameters demonstrated.
Study minimax-optimal rates for offline decision-making with function approximation.
problem Statistical complexity of offline decision-making with function approximation.
method Near minimax-optimal rates for stochastic contextual bandits and Markov decision processes, using pseudo-dimension and behavior policy.
result Established performance limits and new characterization of behavior policy.
Optimal persuasion involves projecting state vectors onto lower-dimensional 'optimal information manifolds'.
problem Optimal persuasion of another agent observing multi-dimensional data.
method Performing non-linear dimension reduction by projecting state vectors onto the 'optimal information manifold'.
result Optimal information design splits information into 'good' and 'bad' components, revealing only the direction of good information.
A new pricing strategy maximizes revenue in high-dimensional product spaces with varying customer preferences.
problem Maximizing revenue in a high-dimensional product space with heterogeneous price sensitivity.
method Proposes M3P, a pricing policy that achieves a specific regret bound under heterogeneous price sensitivity.
result Achieves a T T T -period regret of O ( log ( T d ) ( T + d log ( T ) ) ) O(\log(Td) (\sqrt{T} + d\log(T))) O ( log ( T d ) ( T + d log ( T ))) . LEARN-SAM improves RL from sub-optimal demonstrations by localizing expert policies and selectively using demonstrations.
problem Improving RL from sub-optimal or sparse demonstrations.
method Local Ensemble and Reparameterization with Split and Merge of expert policies (LEARN-SAM).
result LEARN-SAM boosts learning speed and accuracy by selectively using demonstrations.
Study shows model-based methods require fewer samples than model-free methods for LQR tasks.
problem Comparing model-based and model-free methods in reinforcement learning for continuous control tasks.
method An asymptotic analysis of sample complexity for policy evaluation in LQR tasks.
result Model-based methods require asymptotically less samples than model-free methods for policy evaluation in LQR tasks.
Q-MMR evaluates policies using reweighted rewards and moment matching.
problem Off-policy evaluation in finite-horizon MDPs.
method Q-MMR learns scalar weights for data points via a moment matching objective against a value-function discriminator class.
result Data-dependent finite-sample guarantee with a dimension-free error bound.
Deep neural networks can estimate Q-values efficiently on low-dimensional state-action spaces.
problem Estimating the performance of a reinforcement learning policy using data from a different policy.
method Deep fitted Q-evaluation method leveraging manifold structure and convolutional neural networks.
result Sharp error bound for fitted Q-evaluation depends on intrinsic dimension and function space mismatch.
NPMD uses CNNs to optimize policies on low-dimensional manifolds, reducing sample complexity.
problem Explaining the effectiveness of deep policy gradient methods in high-dimensional RL.
method Neural policy mirror descent (NPMD) with CNNs, considering state spaces as low-dimensional manifolds.
result NPMD finds ε-optimal policies with O(ε^(-d/α-2)) samples, leveraging low-dimensional structure.
A scalable framework optimizes multi-asset portfolios with constraints.
problem Optimizing multi-asset portfolios with inequality constraints.
method Integrates neural policies with Pontryagin's Maximum Principle, enforcing feasibility via log-barrier regularization.
result Recover KKT-optimal policies in high-dimensional problems without violating constraints.
MA-COPP predicts multi-agent system outcomes using data from a different policy, with probabilistic guarantees.
problem Predicting outcomes in multi-agent systems using data from a different policy.
method Conformal prediction framework applied to multi-agent systems, avoiding exhaustive search.
result Achieves probabilistic guarantees for multi-agent system predictions.
The UN General Debate Corpus analyzes speeches from UN member states to reveal their political positions.
problem Lack of data on state preferences in international politics.
method Text analysis of over 7,700 speeches from 1970-2016.
result Demonstrates how the UN General Debate Corpus can reveal country positions on various policy dimensions.
This paper improves RL for PDE control with function-valued actions.
problem Control of PDEs with high-dimensional, spatially related actions.
method Action descriptors and deep deterministic policy gradient.
result Action descriptor approach is more sample efficient.
New algorithms identify best policies in discounted linear MDPs efficiently.
problem Identifying the best policy in discounted linear MDPs with limited samples.
method Derive lower bounds and devise simple yet near-optimal algorithms.
result Upper bound on sample complexity matches existing bounds.
This paper tackles data-efficient nonlinear control in Hamiltonian systems using symplectic geometry.
problem Data-efficient nonlinear control in Hamiltonian systems.
method Combines symplectic geometry, recurrence on energy level sets, and chain policies to solve target reachability problems.
result Data requirements depend on geometric and recurrence properties of the Hamiltonian, not the state dimension.
SNAPO optimizes policies for complex sequential decisions using differentiable simulation.
problem Optimizing policies for high-dimensional, sequential decisions under uncertainty.
method Embeds neural policy in a differentiable simulator, computes gradients efficiently.
result Produces sensitivities at a cost proportional to one reverse pass, regardless of sensitivity count.
RL solves large-scale MV portfolio allocation with high returns.
problem Large-scale mean-variance portfolio optimization.
method Continuous-time reinforcement learning with a multivariate Gaussian policy.
result Our method outperforms econometric and deep RL methods by significant margins.
This paper identifies drift Lipschitz budget K as key to diffusion policy expressivity and statistical trade-offs.
problem Understanding and maximizing the expressivity of diffusion policies while managing statistical limitations.
method Identifying drift Lipschitz budget K as central, quantifying expressivity and statistical behavior, proving lower bounds, and providing practical implementation guidelines.
result Balancing expressivity and statistical complexity yields a finite-sample performance gap, with rates depending on sample size and drift type.
Efficient local planning with linear approximations for agents with limited simulator access.
problem Planning with limited simulator access in reinforcement learning.
method Confident Monte Carlo Least Square Policy Iteration (Confident MC-LSPI) and Politex (Confident MC-Politex) algorithms.
result The algorithms can learn the optimal policy with local simulator access, even for linear Q-functions.