HiDe learns hierarchical control for complex tasks by separating planning and control.
problem Solving long horizon control tasks with generalization to unseen scenarios.
method Functional decomposition of state-action spaces, RL-based planner, modular transfer of policy layers.
result Generalizes across unseen test environments and scales to longer horizons.
AutoRL automates reward tuning for RL, improving performance on complex tasks.
problem Suboptimal policies when using direct rewards in RL.
method Evolutionary search over RL agents to find optimal rewards.
result Improves RL performance on complex tasks.
Reliable and effective multi-task learning is a prerequisite for the development of robotic agents that can quickly learn to accomplish related, everyday tasks. However, in the reinforcement learning domain, multi-task learning has not exhibited the same level of success as in other domains, such as computer vision. In…
Tabular Q-Learning with learned state abstractions solves continuous control tasks.
problem Challenging reinforcement learning problems in continuous control.
method Learned state abstraction to transform continuous state-space into discrete.
result Tabular Q-Learning with learned abstractions achieves efficient learning in unseen tasks.
Automatically finds strong neural network topologies for continuous control tasks.
problem Handcrafted neural network architectures limit the performance of Deep Reinforcement Learning.
method Combines Neuroevolution with off-policy training and proposes a novel architecture mutation operator.
result The proposed Actor-Critic Neuroevolution algorithm often outperforms strong baseline methods.
This work evaluates task-agnostic exploration methods for fixed-batch learning.
problem Expensive real-world experience for robotics tasks.
method Fixed datasets for arbitrary task learning.
result Improved offline learning for robotics tasks.
Weakly-supervised RL identifies meaningful tasks, improving performance in complex environments.
problem Learning to efficiently explore and distinguish between meaningful and irrelevant tasks.
method Weak supervision to automatically disentangle meaningful tasks from a large space of nonsensical tasks.
result The learned subspace of meaningful tasks leads to substantial performance gains, especially in complex environments.
Paper tackles continual reinforcement learning challenges with diversity exploration and adversarial self-correction.
problem Challenges in learning new tasks sequentially due to catastrophic forgetting.
method Develops an end-to-end framework (CDAN) combining unsupervised diversity exploration and adversarial self-correction.
result Final result outperforms baseline by 18.35% in NSD and 0.61 in average reward.
CLEAS improves neural architecture search for continual learning.
problem Overcoming catastrophic forgetting and adapting to new tasks while controlling model complexity.
method Neural architecture search (NAS) with reinforcement learning to find optimal neural architecture.
result CLEAS achieves higher classification accuracy with simpler neural architectures.
New model learning objective improves continuous control tasks.
problem Challenges in solving continuous control tasks using model-based reinforcement learning.
method Derived a novel value-aware model learning objective and identified and addressed stale value estimates issue.
result Value-aware objectives can be successfully deployed in solving continuous control tasks without tuning hyper-parameters.
Reinforcement learning algorithms rely on exploration to discover new behaviors, which is typically achieved by following a stochastic policy. In continuous control tasks, policies with a Gaussian distribution have been widely adopted. Gaussian exploration however does not result in smooth trajectories that generally c…
Continuous-time MBRL framework tackles control systems with Bayesian ODEs.
problem Discretization of continuous-time systems in MBRL.
method Novel actor-critic method with Bayesian ODEs for state inference.
result Model robust against irregular and noisy data, sample-efficient, solves challenging control problems.
A new method improves stability in policy learning for continuous control tasks.
problem Stability issues in policy gradient methods when policies are close to deterministic.
method Target Distribution Learning (TDL) alternates between proposing a target distribution and training the policy network to approach it.
result TDL leads to more stable policy improvements over iterations compared to existing methods.
FiDi-RL combines deep RL with FiDi policy search for efficient continuous control.
problem Challenges in continuous control tasks using reinforcement learning.
method Combines Deep Deterministic Policy Gradients (DDPG) with Augment Random Search (ARS) using FiDi policy search.
result Improves data efficiency and stability of ARS.
Faster policy learning via continuous-time gradients.
problem Efficiently estimating policy gradients for continuous-time systems.
method Approximating continuous-time gradients directly, using adaptive discretization.
result More efficient policy gradient estimator leads to faster learning.
Learning when to communicate and doing that effectively is essential in multi-agent tasks. Recent works show that continuous communication allows efficient training with back-propagation in multi-agent scenarios, but have been restricted to fully-cooperative tasks. In this paper, we present Individualized Controlled Co…
DyNODE uses neural ODEs to model system dynamics in continuous control tasks.
problem Modeling the dynamics of systems in continuous control tasks.
method Neural Ordinary Differential Equations (ODEs) combined with actor-critic RL.
result DyNODE outperforms standard neural networks in sample efficiency and predictive performance.
Deep learning solves complex stochastic control with jumps.
problem Solving high-dimensional stochastic control tasks with jumps.
method Model-based approach using two neural networks, iteratively trained with objectives derived from the Hamilton-Jacobi-Bellman equation.
result Demonstrates effectiveness in solving complex high-dimensional stochastic control tasks.
Policy optimization on high-dimensional continuous control tasks exhibits its difficulty caused by the large variance of the policy gradient estimators. We present the action subspace dependent gradient (ASDG) estimator which incorporates the Rao-Blackwell theorem (RB) and Control Variates (CV) into a unified framework…
Paper tackles continual reinforcement learning by forgetting, proposing a planning method with online world models.
problem Catastrophic forgetting in reinforcement learning when learning new tasks.
method Planning with an online world model using model predictive control.
result The proposed FTL Online Agent (OA) learns new tasks without forgetting old skills.
Many continuous control tasks have bounded action spaces. When policy gradient methods are applied to such tasks, out-of-bound actions need to be clipped before execution, while policies are usually optimized as if the actions are not clipped. We propose a policy gradient estimator that exploits the knowledge of action…
We propose a method for tackling catastrophic forgetting in deep reinforcement learning that is \textit{agnostic} to the timescale of changes in the distribution of experiences, does not require knowledge of task boundaries, and can adapt in \textit{continuously} changing environments. In our \textit{policy consolidati…
USAC balances pessimism and optimism in actor-critic training for better exploration and performance.
problem Excessive pessimism limits exploration, while excessive optimism leads to high-risk behaviors.
method Utility Soft Actor-Critic (USAC) dynamically adapts exploration based on critic uncertainty.
result USAC consistently outperforms state-of-the-art algorithms in continuous control tasks.
New self-imitation learning method improves performance in continuous control tasks.
problem Improving off-policy learning in continuous control tasks.
method Proposes a n-step lower bound to generalize lower-bound Q-learning and introduces a new family of self-imitation learning algorithms.
result n-step lower bound Q-learning achieves a better trade-off between bias and contraction rate, leading to improved performance.
Computational Drug Repositioning (CDR) is the task of discovering potential new indications for existing drugs by mining large-scale heterogeneous drug-related data sources. Leveraging the patient-level temporal ordering information between numeric physiological measurements and various drug prescriptions provided in E…
A new RL method uses flows to mix policies for better exploration.
problem Learning in high-dimensional state spaces with deceptive rewards.
method Uses normalizing flows to attract and repel policies in a population.
result Empirically outperforms Soft-Actor Critic (SAC) on MuJoCo tasks.
Nonlinear optimal control problems are often solved with numerical methods that require knowledge of system's dynamics which may be difficult to infer, and that carry a large computational cost associated with iterative calculations. We present a novel neurobiologically inspired hierarchical learning framework, Reinfor…
Paper establishes baselines for offline RL from visual observations.
problem Challenges in offline reinforcement learning from visual observations with continuous action spaces.
method Simple baselines and benchmarking tasks for offline RL from visual observations.
result Simple modifications to existing online RL algorithms outperform existing offline RL methods.
ACSSM models irregular time series with continuous dynamics.
problem Modeling irregular time series data.
method ACSSM uses a multi-marginal Doob's h-transform and variational inference with stochastic optimal control.
result ACSSM outperforms in tasks like classification, regression, interpolation, and extrapolation.
Residual Continual Learning prevents forgetting in sequential tasks.
problem Preventing catastrophic forgetting in sequential learning of multiple tasks.
method ResCL reparameterizes network parameters by combining original and fine-tuned networks, keeping network size constant.
result ResCL achieves state-of-the-art performance in various continual learning scenarios.
Inverse optimal control, also known as inverse reinforcement learning, is the problem of recovering an unknown reward function in a Markov decision process from expert demonstrations of the optimal policy. We introduce a probabilistic inverse optimal control algorithm that scales gracefully with task dimensionality, an…
Action chunking and data exploration improve behavior cloning in robotics.
problem Exponential errors in learning from demonstrations for continuous control tasks.
method Action chunking and exploratory data collection.
result Control-theoretic stability is key to improving imitation learning.
New method improves policy gradient performance in continuous control tasks.
problem Improving policy gradient methods for continuous control tasks.
method Numerical integration approach to all-action policy gradient.
result Improved performance and sample efficiency in continuous control tasks.
Study uses DRL with Lagrangian relaxation to solve temporal control tasks with STL constraints.
problem Optimal control problems with temporal logic constraints.
method Extended CMDP formulation, Lagrangian relaxation, two-phase constrained DRL algorithm.
result Demonstrated learning performance of the proposed algorithm through simulations.
Paper proposes a method to improve off-policy reinforcement learning in batch settings.
problem Challenges in applying off-policy reinforcement learning to batch data.
method Uses a learned prior, the advantage-weighted behavior model (ABM), to bias RL policies.
result Improves performance on various RL tasks, including robot control.
Robots learn to handle complex tasks creatively using DRL.
problem Robotic manipulation challenges and intelligence.
method Designing challenging manipulation tasks, applying DRL for robot training.
result Robots exhibit creative and non-intuitive problem-solving.
Paper introduces a meta-critic for accelerating off-policy actor-critic learning.
problem Improving sample efficiency in continuous control tasks.
method Meta-critic that meta-learns an additional loss for the actor.
result Online meta-critic learning leads to improved performance in various continuous control environments.
Algorithm improves continuous control with context, achieving provable efficiency.
problem Building an agent that generalizes to unseen environments in continuous control.
method Uses upper confidence bound (UCB) to refine decoder estimates and balance exploration-exploitation.
result Proves $\widetilde{O}\left(\sqrt{T}
ight)$ regret bound and ε ε ε -suboptimal policy transfer. Study optimal control in unknown nonlinear systems with near-optimal regret bound.
problem Sequential control in unknown, nonlinear dynamical systems.
method LC^3 algorithm, based on information theory.
result Near-optimal O ( T ) O(\sqrt{T}) O ( T ) regret bound for episodic settings. New sensor placement affects robot controller learnability.
problem Catastrophic forgetting in neural controllers for robots.
method Demonstrated how sensor placement alters loss function manifolds.
result Sensor placement can reduce or induce catastrophic forgetting.
In dynamic environments, learned controllers are supposed to take motion into account when selecting the action to be taken. However, in existing reinforcement learning works motion is rarely treated explicitly; it is rather assumed that the controller learns the necessary motion representation from temporal stacks of …
RP1 uses active learning to improve world model in fewest samples.
problem Improving sample efficiency in MBRL for continuous control tasks.
method RP1 views MBRL as active learning, using a hybrid objective function and principled termination.
result Statistically significant gains over existing approaches on continuous control tasks.
INDEQS: A Graph-Based Neural Controlled Differential Equation Framework for Forecasting
problem Forecasting time series with neural networks
method Incorporating prior knowledge of a directed graph
result Outer informedness consistently improves forecasting accuracy
Study bank salvage model with stochastic impulse controls to minimize costs.
problem Minimize total cost of saving a bank from default with unpredictable default time.
method Impulse stochastic controls to address the bank's default risk.
result Unique viscosity solution exists for the QVI, with Lipschitz and Holder continuity properties.
Novel method controls complex physical systems over long time frames.
problem Controlling complex nonlinear physical systems over long time frames.
method Hierarchical predictor-corrector scheme with separate planning and control networks.
result Successfully controls complex physical systems like incompressible Navier-Stokes equations.
Robot learns multiple tasks hierarchically by transferring knowledge.
problem Learning multiple complex tasks in open-ended environments.
method Task-oriented procedures, goal-babbling, imitation learning, active learning, intrinsic motivation.
result Robots can learn complex tasks more efficiently by transferring knowledge from simpler ones.
PFPN uses particle filtering to improve character control in physics-based simulations.
problem Premature commitment to suboptimal actions in high-dimensional continuous control problems for articulated characters.
method Proposes a particle-based action policy using particle filtering to dynamically explore and discretize the action space.
result Demonstrates better imitation performance and robustness to external perturbations compared to Gaussian policies.
Proposes a Quasi-Newton trust region method for policy optimization in reinforcement learning.
problem Lack of stepsize selection criterion and slow convergence in gradient descent for policy optimization.
method Uses a trust region method with Quasi-Newton approximation for the Hessian.
result Demonstrates improved performance and efficiency in continuous control tasks.