Survey explores geometric aspects of policy optimization in control systems.
problem Understanding the geometric relationships between control design and optimization.
method Geometric perspective on policy optimization, focusing on parameterization and topology.
result Implications of policy geometry on stability and performance of local search algorithms.
Adapts model-based advice to stabilize black-box policies for nonlinear control.
problem Stabilizing machine-learned policies for nonlinear control with limited model information.
method Proposes an adaptive λ-confident policy to combine black-box and model-based advice. result Proves the stability of the adaptive λ-confident policy and its competitive ratio. PFPN uses particle filtering to improve character control in physics-based simulations.
problem Premature commitment to suboptimal actions in high-dimensional continuous control problems for articulated characters.
method Proposes a particle-based action policy using particle filtering to dynamically explore and discretize the action space.
result Demonstrates better imitation performance and robustness to external perturbations compared to Gaussian policies.
New RL framework learns optimal control policies directly.
problem Directly learning optimal control policies in complex systems.
method General Markov Decision Process (MDP) framework for reinforcement learning.
result Convergence and optimality of control-based RL methods established.
Paper proposes a new method to optimize robot body structure and control policy.
problem Optimizing robot body structure and control policy in a coupled manner.
method Revisits co-design problem as a Stackelberg game, incorporating control adaptation dynamics.
result Stackelberg PPO outperforms standard PPO in stability and performance.
New framework for policy gradient methods in continuous time reinforcement learning.
problem Addressing policy gradient methods for continuous time reinforcement learning.
method Control randomisation technique to derive policy gradient representation for various Markovian control problems.
result Demonstrated application to optimal switching problems in the energy sector.
Survey of theoretical foundations for policy optimization in control.
problem Understanding the theoretical properties of gradient-based methods in control and reinforcement learning.
method Interdisciplinary review of optimization landscape, convergence, and sample complexity for various control problems.
result Recent theoretical results on stability and robustness in learning-based control.
New RL algorithms improve control tasks with data reuse.
problem Real-world control requires performance guarantees and data efficiency.
method Generalized Policy Improvement combining on-policy guarantees and sample reuse.
result Extensive experimental analysis shows benefits of new algorithms.
Researchers develop methods to protect reinforcement learning and control systems from policy poisoning attacks.
problem Attacks on reinforcement learning and control systems that manipulate learned policies.
method Unified framework for solving policy poisoning attacks, demonstrating global optimality and feasibility.
result Policy poisoning attacks are feasible and can be defended with a convex optimization approach.
Paper optimizes traffic signal control for better traffic flow.
problem Optimizing traffic signal control to reduce congestion and improve safety.
method Value-based reinforcement learning with interpretable policy functions (polynomial functions).
result Deep Regulatable Hardmax Q-learning variant reduces vehicle delay by up to 19.4%.
New method transfers robotic control policies to unknown environments using a family of policies.
problem Transfer of robotic control policies trained in simulation to real hardware is difficult due to environment differences.
method Simultaneously learns a family of policies with different behaviors; searches for the best policy based on task performance.
result Demonstrates superior performance in unknown environments compared to other methods, overcoming larger modeling errors.
The paper examines how macroeconomic control tools lost effectiveness, leading to a 'dark ages' period.
problem Loss of effectiveness of control tools in macroeconomic stabilization policy.
method Historical analysis of macroeconomic stabilization policy from 1948 to 1993.
result The overstatement of the Lucas critique and Kydland and Prescott's time-inconsistency led to a period of ineffective stabilization policy.
A new policy gradient estimator reduces variance for clipped actions in continuous control tasks.
problem Policy gradient methods struggle with bounded action spaces.
method Proposes a new policy gradient estimator that accounts for clipped actions.
result The new estimator achieves lower variance and outperforms conventional methods.
New method reduces variance in policy gradient estimation for reinforcement learning.
problem Large variance issue in policy gradient estimation.
method Action-dependent control variates using Stein's identity.
result Significantly improves sample efficiency of policy gradient methods.
A new method improves stability in policy learning for continuous control tasks.
problem Stability issues in policy gradient methods when policies are close to deterministic.
method Target Distribution Learning (TDL) alternates between proposing a target distribution and training the policy network to approach it.
result TDL leads to more stable policy improvements over iterations compared to existing methods.
Meta-reinforcement learning improves fault-adaptive control efficiency.
problem Adaptive control under abrupt system faults with strict time constraints.
method Model-agnostic meta learning (MAML) with a fault library of prior policies.
result Improved sample efficiency and quick adaptation to new faults.
MVPI framework optimizes risk in reinforcement learning, improving performance in robot simulations.
problem Optimizing risk in reinforcement learning control problems.
method Mean-Variance Policy Iteration (MVPI) framework for risk-averse control in MDPs.
result Risk-averse TD3 outperforms previous methods in robot simulation tasks.
Novel framework learns policies for MPC with near-optimal guarantees.
problem Safe and near-optimal policy learning for MPC.
method Primal-dual neural networks for supervised learning of policies and certificate policies.
result Guaranteed feasibility and optimality of learned policies.
Trust-region methods and natural gradients are equivalent in certain policy search scenarios.
problem Improving policy search methods in continuous control tasks.
method Introducing compatible policy search (COPOS) that uses natural parameterization and compatible value function approximation to control entropy loss.
result COPOS yields state-of-the-art results in challenging tasks and reduces entropy loss.
This work ensures policy gradient methods converge to global optima for certain control problems.
problem Non-convex optimization challenges in policy gradient methods for complex control problems.
method Identifies structural properties ensuring non-convex objective functions have no suboptimal stationary points.
result Policy gradient methods converge to global optima under certain conditions, satisfying a Polyak-Lojasiewicz condition.
Developed policy gradient methods for stochastic control with exit time, outperforming traditional techniques in share repurchase pricing.
problem Optimal control with exit time in stochastic models.
method Two types of algorithms: direct policy learning and alternately learning value function and control.
result Policy gradient methods outperform PDE or neural networks in share repurchase pricing.
The paper develops a method to create robust control policies for robots using information bottlenecks.
problem Robotic control policies are sensitive to task-irrelevant state and sensor changes.
method Derives a policy gradient algorithm that creates an information bottleneck between states and task-relevant representations.
result Task-driven policies are more robust to sensor noise and environmental changes.
Efficient deep policy gradient method for continuous-time control problems.
problem Optimal control in continuous time with fine time discretization.
method Multi-scale deep policy gradient method with varying time discretization.
result Targeted efficiency in computational resources achieved through multi-scale approach.
A contextual bandit method evaluates and improves inventory control policies.
problem Evaluating and improving periodic review inventory control policies with nonstationary demand.
method Contextual bandit-based algorithm to evaluate and tweak policies.
result The method achieves favorable guarantees in both theory and practice.
The paper proposes a new method to estimate optimal policies using MCMC.
problem Estimating the optimal policy for systems with unknown dynamics and reward functions.
method Using Markov Chain Monte Carlo to generate samples from the posterior distribution of parameters conditioned on optimality.
result The method provably converges to the globally optimal stochastic policy with similar variance to policy gradient methods.
Researchers solve a complex financial control problem with explicit policies.
problem Constrained LQ control with multiplicative noise in financial risk management.
method Derived analytical control policy using state separation property and solving coupled Riccati equations.
result Explicit piece-wise affine control policy for optimal control of stochastic systems.
This paper tackles robust control of LQR systems with multiplicative noise using policy gradient methods.
problem Robustness in reinforcement learning control of complex systems with multiplicative noise.
method Policy gradient algorithms with gradient domination property for non-convex cost functions.
result Global convergence of policy gradient algorithms to the globally optimum control policy.
PID control architectures are widely used in industrial applications. Despite their low number of open parameters, tuning multiple, coupled PID controllers can become tedious in practice. In this paper, we extend PILCO, a model-based policy search framework, to automatically tune multivariate PID controllers purely bas…
Optimizes dividend policies in a Brownian model with controlled rates.
problem Realistic optimal dividend policies in a stochastic control problem.
method Delayed linear control strategies for refracted diffusion processes.
result Optimality of delayed linear control strategies for dividend payments.
Optimal policy for multi-hypothesis testing with controlled sensing to minimize delay and error.
problem Minimizing delay in multi-hypothesis testing with controlled sensing.
method Designing a policy to control the delay while ensuring error probability constraint.
result Policy achieves information-theoretic lower bound on expected delay asymptotically.
Paper develops a distributed power control method for large energy harvesting networks using deep reinforcement learning.
problem Optimal power control for large energy harvesting networks with limited causal information.
method Multi-agent reinforcement learning framework to solve a mean-field game problem.
result Proposed method converges to optimal power control policies in a distributed fashion.
Unified policy controls diverse agents through modular neural networks.
problem Learning control policies for various agent morphologies.
method Shared Modular Policies (SMP) with decentralized control and message passing.
result A single modular policy controls multiple agent morphologies.
Paper uses imitation learning to create efficient insulin policies from MPC demonstrations.
problem Resource-constrained medical devices struggle with complex MPC optimizations and state estimation errors.
method Imitation learning of neural network policies from MPC-computed demonstrations, using Bayesian inference with Monte Carlo Dropout.
result Trained policies generalize well to different patient cohorts, outperforming traditional MPC with state estimation.
A new method for continuous control avoids local movement issues.
problem Limitations of policy gradient methods in continuous control.
method Distributional framework and Generative Actor Critic (GAC) method.
result GAC outperforms policy gradient methods in continuous domains.
Automated testing framework finds weaknesses in deep control policies.
problem Safety of deep neural network control policies is difficult to validate.
method Adversarial reinforcement learning to test and find weaknesses.
result Framework finds weaknesses not evident during manual testing.
New method reduces over-pessimism in Bayesian control under parameter uncertainty.
problem Over-pessimism in Bayesian control due to misspecified priors.
method Distributionally robust Bayesian control (DRBC) with strong duality and optimization.
result Validated algorithm on synthetic and real data, reducing over-pessimism.
Method trains vision and control policies on real robots quickly.
problem Training vision-based control policies on real robots efficiently.
method Multi-task Reinforcement Learning with auxiliary tasks.
result Significant learning speed-ups and task learning from-scratch.
Risk-controlled post-processing optimizes decision policies under risk constraints.
problem Optimizing decision policies with risk constraints for better outcomes.
method Developed a post-processing algorithm that selects a threshold based on fitted fallback policy and score, leveraging tools from algorithmic stability and stochastic processes.
result The post-processed policy achieves precise expected risk control under exchangeability and meets or nearly meets risk budgets while preserving more agreement with the baseline.
Study nonparametric estimator for Markov chain transition matrices in offline setting.
problem Estimating transition matrices of finite controlled Markov chains from logged data.
method Developed sample complexity bounds and conditions for minimaxity.
result Achieving certain statistical risk requires balancing mixing properties and sample size.
Develops a new reinforcement learning framework for complex control problems.
problem Continuous-time extended mean field control with deterministic policies.
method Model-free sensitivity formula, deterministic policy gradient, local value and advantage-rate representations.
result Demonstrates efficiency, stability, and robustness in solving complex control problems.
New approach to understand recurrent policies as FSMs without minimization.
problem Minimization of FSMs obscures the semantics of policy decisions.
method Start with unminimized FSM, apply interpretable reductions, use attention tool.
result Reveals insights into policy decisions not previously noticed.
Optimizes control of noisy discrete systems without system matrix knowledge.
problem Optimal control of discrete-time systems with additive and multiplicative noises.
method Stochastic Lyapunov and Riccati equations, model-free reinforcement learning.
result Model-free reinforcement learning algorithm converges to optimal control policy.
Framework learns robust control policies from expert demonstrations.
problem Adversarial robustness and closed-loop generalization in feedback control policies.
method Lipschitz-constrained loss minimization for certified robustness and generalization.
result Finite sample bound on policy learning error and robust closed-loop stability.
RL solves discrete LQ control with Gaussian optimal policy.
problem Discrete-time linear-quadratic control problem.
method Entropy-based RL to find Gaussian optimal policy.
result RL algorithm solves mean-variance asset-liability management problem.
Proposes a model-free control method for chaotic systems using deep Q-learning.
problem Chaos control in systems without precise mathematical models.
method Data-based control policy with two steps: region determination and policy learning.
result Efficient control policy learning in a limited region of state space.
The paper presents a model-free method for stabilizing unknown control systems.
problem Stabilizing unknown control systems in engineering.
method Solving discounted LQR problems with increasing discount factors.
result The method efficiently recovers a stabilizing controller for linear and smooth nonlinear systems.
New algorithm reduces decision switching in dynamic environments.
problem Online learning with memory and non-stationary environments.
method Dynamic policy regret, novel ensemble approach, meta-base decomposition.
result Proves optimal dynamic policy regret for memory length, non-stationarity, and time horizon.
Paper formulates mutual information optimal control for discrete-time systems.
problem Optimal control of discrete-time linear systems with mutual information.
method Formulates MIOCP as an extension of MEOCP, derives optimal policy and prior, proposes alternating minimization algorithm.
result Proposes an alternating minimization algorithm for MIOCP.