Study on adversarial training's impact on deep neural reinforcement learning policies.
problem Vulnerability of deep neural reinforcement learning policies to imperceptible adversarial perturbations.
method Two parallel approaches: Fourier spectrum analysis and feature sensitivity measurement.
result Adversarially trained policies are more sensitive to low frequency perturbations.
Policy gradient methods with actor-critic schemes demonstrate tremendous empirical successes, especially when the actors and critics are parameterized by neural networks. However, it remains less clear whether such "neural" policy gradient methods converge to globally optimal policies and whether they even converge at …
We present a reinforcement learning framework, called Programmatically Interpretable Reinforcement Learning (PIRL), that is designed to generate interpretable and verifiable agent policies. Unlike the popular Deep Reinforcement Learning (DRL) paradigm, which represents policies by neural networks, PIRL represents polic…
Softmax policy gradient achieves global optimality in wide neural networks with entropy regularization.
problem Optimizing softmax policies with neural networks in the mean-field regime.
method Modeling neural networks as Wasserstein gradient flows and proving global optimality of fixed points.
result Global optimality of softmax policy gradient in wide single hidden layer neural networks with entropy regularization.
Paper analyzes neural network complexity for planning problems.
problem Understanding neural network complexity for planning policies.
method Circuit complexity analysis for relational neural networks.
result Three classes of planning problems identified based on network complexity.
Generative neural nets learn deep policies conditioned on goals.
problem Learning optimal policies for specific goals in reinforcement learning.
method Goal-conditioned neural nets that generate deep neural policies.
result Single learned policy generator can achieve any desired return.
Proximal policy optimization and trust region policy optimization (PPO and TRPO) with actor and critic parametrized by neural networks achieve significant empirical success in deep reinforcement learning. However, due to nonconvexity, the global convergence of PPO and TRPO remains less understood, which separates theor…
NeuPL learns diverse policies in strategy games efficiently.
problem Iterative training of policies in strategy games leads to under-trained good-responses and wasteful repetition.
method NeuPL uses a single conditional model to represent a population of policies, offering convergence guarantees and transfer learning.
result NeuPL achieves better performance and efficiency across various domains, enabling access to novel strategies.
Paper tackles DOCTR-L with SciPhy RL, solving neural PDEs from data.
problem High-dimensional optimal control with stochastic policies.
method Soft HJB equation, Neural PDEs, Physics-Informed Neural Networks.
result Reduces DOCTR-L to solving neural PDEs from data.
Revel tackles safe exploration in RL with verified symbolic policies.
problem Computational infeasibility of verifying neural networks in RL learning loops.
method Two policy classes: neurosymbolic with approximate gradients and symbolic policies for efficient verification. Mirror descent over policies to safely update and project policies.
result Revel discovers policies that outperform prior approaches to verified exploration.
SymNet uses neural models to solve RMDPs efficiently.
problem Limited effectiveness of generalized policies in RMDPs.
method SymNet trains shared parameters for an RDDL domain, converts instances to graphs, uses relational neural models for node embeddings, and scores actions based on embeddings.
result SymNet policies outperform random and state-of-the-art deep reactive policies on nine RDDL domains.
We compare the model-free reinforcement learning with the model-based approaches through the lens of the expressive power of neural networks for policies, Q-functions, and dynamics. We show, theoretically and empirically, that even for one-dimensional continuous state space, there are many MDPs whose optimal Q-func…
NDPs embed dynamical systems into neural networks for efficient sensorimotor learning.
problem Training policies directly in raw action spaces limits scalability for continuous tasks.
method Embed dynamical systems into neural networks to learn robot behaviors via demonstrations.
result NDPs outperform prior methods in both imitation and reinforcement learning setups.
Paper analyzes NAC with neural networks for efficient policy optimization.
problem Improving sample and iteration complexity in policy optimization.
method Entropy regularization, averaging, neural network approximation, and optimization techniques.
result Entropy regularization and averaging ensure stability and sharp sample complexity bounds.
Cyclical learning rate improves neural machine translation performance.
problem Optimizing learning rate for neural machine translation.
method Applied cyclical learning rate to transformer-based neural networks.
result Cyclical learning rate significantly impacts neural machine translation performance.
NPMD uses CNNs to optimize policies on low-dimensional manifolds, reducing sample complexity.
problem Explaining the effectiveness of deep policy gradient methods in high-dimensional RL.
method Neural policy mirror descent (NPMD) with CNNs, considering state spaces as low-dimensional manifolds.
result NPMD finds ε-optimal policies with O(ε^(-d/α-2)) samples, leveraging low-dimensional structure.
Optimizes insurance pricing by accounting for policyholders' price sensitivity.
problem Traditional insurance pricing does not consider policyholders' price sensitivity.
method Formulates insurance pricing as a decision-making problem and uses off-policy evaluation and stochastic control.
result Neural networks outperform existing techniques for policy optimization.
This work learns exploration policies for unknown distributions using samples and policy gradients.
problem Learning exploration policies for unknown distributions in Bayesian bandits.
method Meta-learning approach parameterizing policies in a differentiable way and optimizing them with policy gradients.
result Effective gradient estimators and variance reduction techniques are derived.
Deep RL policies share adversarial features across different MDPs.
problem Understanding decision boundaries and loss landscapes in neural policies.
method Investigating similarities in high sensitivity directions across MDPs using Arcade Learning Environment.
result High sensitivity directions for neural policies are correlated across MDPs, suggesting shared non-robust features.
Approach to optimize bidding policies offline using reinforcement learning.
problem Optimizing spending in online advertising under budget constraints.
method Offline reinforcement learning for optimizing differentiable base policies.
result Statistically significant performance gains in production bidding environments.
Solves POMDPs with recurrent neural networks and natural policy gradient.
problem Non-stationarity in optimal policies of POMDPs.
method Integrates recurrent neural networks into natural policy gradient and temporal difference learning.
result Non-asymptotic theoretical guarantees for global optimality up to function approximation.
Deep neural networks can estimate Q-values efficiently on low-dimensional state-action spaces.
problem Estimating the performance of a reinforcement learning policy using data from a different policy.
method Deep fitted Q-evaluation method leveraging manifold structure and convolutional neural networks.
result Sharp error bound for fitted Q-evaluation depends on intrinsic dimension and function space mismatch.
Temporal-difference learning (TD), coupled with neural networks, is among the most fundamental building blocks of deep reinforcement learning. However, due to the nonlinearity in value function approximation, such a coupling leads to nonconvexity and even divergence in optimization. As a result, the global convergence …
Deep RL policies are vulnerable to adversarial perturbations, but vanilla training yields more robust policies.
problem Vulnerability of deep reinforcement learning policies to adversarial perturbations.
method Analysis of deep reinforcement learning policy landscape and comparison of vanilla vs. adversarial training.
result Vanilla training yields more robust policies compared to adversarial training.
Efficient policy learning from observational data using weighted classification reductions.
problem Efficient policy evaluation does not necessarily lead to efficient estimation of policy parameters.
method Proposed an estimation approach based on generalized method of moments, efficient for policy parameters.
result Demonstrated empirical efficiency and regret benefits of a proposed method.
Study how untrained policies explore in RL environments.
problem Challenges in reinforcement learning, especially sparse or adversarial reward structures.
method Theoretical and empirical analysis of untrained deep neural policies in a toy model.
result Untrained policies generate correlated actions and non-trivial state-visitation distributions.
DPDP combines neural heuristics with DP for vehicle routing problems.
problem Vehicle routing problems with large scale.
method Deep Policy Dynamic Programming (DPDP) that uses a neural network policy to prioritize and restrict the DP state space.
result DPDP improves upon classical DP algorithms and outperforms neural approaches for TSP, VRP, and TSPTW.
Neural Index Policy for multi-action bandits with heterogeneous budgets.
problem Real-world settings often involve multiple interventions with heterogeneous costs and constraints, breaking classical assumptions.
method Introduces a Neural Index Policy (NIP) that learns to assign budget-aware indices to arm-action pairs using a neural network and differentiable knapsack layer.
result Empirically achieves near-optimal performance while strictly enforcing heterogeneous budgets and scaling to hundreds of arms.
Deep neural networks are known to be fragile to small adversarial perturbations. This issue becomes more critical when a neural network is interconnected with a physical system in a closed loop. In this paper, we show how to combine recent works on neural network certification tools (which are mainly used in static set…
The paper studies how neural policies can be interpreted using decision trees.
problem Understanding how machine learning controllers make decisions in complex environments.
method The approach involves disentangled representation using decision trees to interpret neural policies.
result The paper shows that disentanglement of learned neural dynamics improves interpretability.
This paper identifies drift Lipschitz budget K as key to diffusion policy expressivity and statistical trade-offs.
problem Understanding and maximizing the expressivity of diffusion policies while managing statistical limitations.
method Identifying drift Lipschitz budget K as central, quantifying expressivity and statistical behavior, proving lower bounds, and providing practical implementation guidelines.
result Balancing expressivity and statistical complexity yields a finite-sample performance gap, with rates depending on sample size and drift type.
We study the problem of programmatic reinforcement learning, in which policies are represented as short programs in a symbolic language. Programmatic policies can be more interpretable, generalizable, and amenable to formal verification than neural policies; however, designing rigorous learning approaches for such poli…
In this paper we target the problem of transferring policies across multiple environments with different dynamics parameters and motor noise variations, by introducing a framework that decouples the processes of policy learning and system identification. Efficiently transferring learned policies to an unknown environme…
Unified policy controls diverse agents through modular neural networks.
problem Learning control policies for various agent morphologies.
method Shared Modular Policies (SMP) with decentralized control and message passing.
result A single modular policy controls multiple agent morphologies.
Lo-Hp decouples weight generation into local and global policies to improve flexibility and efficiency.
problem Over-coupling and long-horizon issues in current optimization methods.
method Hybrid-Policy Sub-Trajectory Balance objective.
result Learning local optimization policies addresses long-horizon issues and enhances global weight generation.
Paper uses imitation learning to create efficient insulin policies from MPC demonstrations.
problem Resource-constrained medical devices struggle with complex MPC optimizations and state estimation errors.
method Imitation learning of neural network policies from MPC-computed demonstrations, using Bayesian inference with Monte Carlo Dropout.
result Trained policies generalize well to different patient cohorts, outperforming traditional MPC with state estimation.
HAMBO estimates policy performance by hallucinating worst-case trajectories, providing valid lower bounds.
problem Conservative off-policy evaluation of policies in real-world applications.
method HAMBO hallucinates worst-case trajectories based on learned model uncertainty.
result Valid lower bounds on policy performance, converging to true expected return under regular conditions.
In this work, we consider the problem of estimating a behaviour policy for use in Off-Policy Policy Evaluation (OPE) when the true behaviour policy is unknown. Via a series of empirical studies, we demonstrate how accurate OPE is strongly dependent on the calibration of estimated behaviour policy models: how precisely …
Unified approach to Merton's portfolio problem using Pontryagin's principles.
problem Optimizing consumption and investment strategies in financial portfolios.
method PG-DPO framework combining neural networks with Pontryagin's maximum principle.
result Locally optimal policies closely tied to classical stochastic control.
Due to the high variance of policy gradients, on-policy optimization algorithms are plagued with low sample efficiency. In this work, we propose Augment-Reinforce-Merge (ARM) policy gradient estimator as an unbiased low-variance alternative to previous baseline estimators on tasks with binary action space, inspired by …
Study shows policy gradient convergence for entropy-regularized MDPs with neural nets in mean-field regime.
problem Global convergence of policy gradient for entropy-regularized MDPs with neural network approximation.
method Softmax policy with neural network approximation in mean-field regime, gradient flow in 2-Wasserstein metric, exponential convergence under sufficient regularization.
result Gradient flow converges exponentially fast to the unique stationary solution under sufficient regularization.
PNNs improve personalized healthcare policies using mixed integer programming.
problem Learning treatment policies for patients with limited data.
method Prescriptive networks (PNNs) trained with mixed integer programming.
result PNNs outperform existing methods in reducing peak blood pressure.
New IRL algorithm identifies optimal reward and policy from expert demonstrations.
problem Understanding reward functions from expert demonstrations with neural networks.
method Two-timescale single-loop IRL algorithm for neural network parameterized rewards.
result First IRL algorithm with non-asymptotic convergence guarantee and global optimality in neural network settings.
Policy-GNN optimizes GNN aggregation for diverse node iterations.
problem Optimizing GNN performance by varying aggregation iterations for different nodes.
method Policy-GNN uses a meta-policy framework with deep reinforcement learning to adaptively determine the number of aggregations for each node.
result Policy-GNN significantly outperforms state-of-the-art alternatives on real-world datasets.
Proposes a method to estimate policy values in reinforcement learning with unmeasured confounders.
problem Estimating policy values in reinforcement learning with unmeasured confounders.
method Develops a two-way deconfounder algorithm using a neural tensor network to learn unmeasured confounders and system dynamics.
result Consistent policy value estimation through model-based estimator.
LALR adapts learning rate for faster convergence in regression and neural nets.
problem Finding optimal learning rates for faster convergence in regression and neural networks.
method Lipschitz continuity theory applied to Mean Absolute Error and Quantile loss functions.
result Adaptive learning rate policy enables up to 20x faster convergence.
Paper proposes interpretable RL policies from a mixture of experts.
problem Making RL policies transparent and understandable in real-world applications.
method Policy iteration scheme with interpretable experts and prototypical states.
result Proposed algorithm learns policies comparable to neural networks but more interpretable.
Develops neural network framework for risk-reward optimization problems.
problem Multi-period risk-reward optimization with constrained policies.
method Neural network framework with two coupled feedforward networks, parametrizing two-step policies.
result Empirical optimum converges to true optimal value as network capacity and training size increase.