Efficiently solves exploration-exploitation in LQR using Lagrangian relaxation.
problem Exploration-exploitation dilemma in linear quadratic regulator (LQR) setting.
method Relax optimistic optimization into a constrained extended LQR problem, then solve using Riccati equations.
result Computes ε ε ε -optimistic controller efficiently with O ( log ( 1 / ε ) ) O\big(\log(1/ε)\big) O ( log ( 1/ ε ) ) Riccati equations. New algorithm learns LQR with O ( T ) O(\sqrt{T}) O ( T ) regret using Langevin dynamics and excitation.
problem Learning LQR with a O ( T ) O(\sqrt{T}) O ( T ) regret bound. method Thompson sampling with Langevin dynamics and excitation mechanism.
result Achieved O ( T ) O(\sqrt{T}) O ( T ) regret bound for LQR learning. Global convergence proved for multi-agent LQRs with hierarchical actor-critic.
problem Challenges in understanding multi-agent reinforcement learning algorithms.
method Developed a hierarchical actor-critic algorithm for partially exchangeable agents.
result Global linear convergence to optimal policy proved.
New model-free algorithm achieves similar LQR regret guarantees.
problem Model-free control of linear dynamical systems under quadratic costs.
method Online policy gradient scheme with policy space cost analysis.
result Achieves regret scaling with √T, matching model-based methods.
Safe RL-based vibration control using LQR guidance.
problem Training risks in RL-based vibration control.
method Hybrid control framework combining LQR and RL.
result LQR controller outperforms uncontrolled scenario.
Study shows PI for LQR requires fewer policy improvement steps than policy evaluation steps.
problem Understanding the sample complexity of RL algorithms for continuous control tasks.
method Finite-time analysis of approximate policy iteration for LQR, quantifying policy improvement and evaluation complexities.
result Policy evaluation is the dominant factor in sample complexity, requiring ( n + d ) 3 / ε 2 (n+d)^3/\varepsilon^2 ( n + d ) 3 / ε 2 samples per step. Policy gradient methods converge for LQR problems with noisy state dynamics.
problem Finding optimal policies in noisy LQR problems over finite time horizons.
method Policy gradient methods with convergence guarantees for finite time and stochastic state dynamics.
result Global linear convergence for policy gradient methods in LQR problems with weak assumptions.
TSAC achieves optimal frequentist regret in adaptive control of LQRs.
problem Adaptive control of stabilizable linear-quadratic regulators with unknown dynamics.
method Thompson Sampling (TS) for adaptive control of LQRs, with a novel early exploration strategy.
result Achieves i l d e O ( T ) ilde O(\sqrt{T}) i l d e O ( T ) regret, optimal for multidimensional systems. We consider adaptive control of the Linear Quadratic Regulator (LQR), where an unknown linear system is controlled subject to quadratic costs. Leveraging recent developments in the estimation of linear systems and in robust controller synthesis, we present the first provably polynomial time algorithm that provides high…
Optimal attack against autoregressive models by manipulating environment states.
problem Manipulating autoregressive forecasts to track a target trajectory.
method Linear Quadratic Regulator (LQR) for linear models, Model Predictive Control (MPC) for nonlinear models.
result Optimal attack formulations for both white-box and black-box settings.
Reinforcement learning (RL) has been successfully used to solve many continuous control tasks. Despite its impressive results however, fundamental questions regarding the sample complexity of RL on continuous problems remain open. We study the performance of RL in this setting by considering the behavior of the Least-S…
Study shows model-based methods require fewer samples than model-free methods for LQR tasks.
problem Comparing model-based and model-free methods in reinforcement learning for continuous control tasks.
method An asymptotic analysis of sample complexity for policy evaluation in LQR tasks.
result Model-based methods require asymptotically less samples than model-free methods for policy evaluation in LQR tasks.
Optimal trading strategy using LQR framework with price mean-reversion.
problem Developing a dynamic trading strategy in a market with linear and quadratic costs.
method Model Predictive Control (MPC) approach to optimize trading curve with positivity constraints.
result Optimal trading curve reacts opportunistically to price changes while satisfying constraints.
SOLAR learns efficient representations for RL in complex image domains.
problem Efficient model-based reinforcement learning in domains with complex observations like images.
method Optimizes structured representations for inferring simple dynamics and cost models from data.
result Substantially better final performance than other model-based RL methods, more efficient than model-free RL.
Paper proposes a COP model for Algo trading using LQR.
problem Complexities in child order placement in Algo trading.
method Stochastic LQR model for passive limit orders and aggressive takeout orders.
result Closed-form solutions for optimal child order placement.
Actor-critic converges globally in LQR with ergodic cost.
problem Theoretical understanding of actor-critic algorithm's global convergence.
method Nonasymptotic convergence analysis of actor-critic in linear quadratic regulator (LQR) setting.
result Actor-critic finds globally optimal policy and value function at a linear rate.
Finding optimal feedback controllers for nonlinear dynamic systems from data is hard. Recently, Bayesian optimization (BO) has been proposed as a powerful framework for direct controller tuning from experimental trials. For selecting the next query point and finding the global optimum, BO relies on a probabilistic desc…
This paper studies how gradient descent in control systems can perform well on unseen data.
problem The extent of a learned controller's ability to extrapolate to unseen initial states.
method Theoretical study of policy gradient in Linear Quadratic Regulator (LQR) problems, focusing on the role of exploration.
result The performance of a learned controller on unseen initial states depends on the degree of exploration induced by the system.
The paper sets bounds on how much regret is unavoidable in adaptive LQR with unknown B-matrix.
problem Understanding the limits of adaptive LQR with unknown B-matrix.
method Local asymptotic minimax regret lower bounds using van Trees' inequality and Bellman error representation.
result Logarithmic regret is impossible if the parametrization induces an uninformative optimal policy.
New bounds for adaptive control in high dimensions without fixed state space.
problem Adaptive control of linear systems in high or infinite dimensions.
method Novel perturbation bound for certainty equivalence, scaling with prediction error.
result First regret bounds for LQR in infinite dimensional systems, independent of ambient dimension.
This paper tackles robust control of LQR systems with multiplicative noise using policy gradient methods.
problem Robustness in reinforcement learning control of complex systems with multiplicative noise.
method Policy gradient algorithms with gradient domination property for non-convex cost functions.
result Global convergence of policy gradient algorithms to the globally optimum control policy.
This manuscript surveys reinforcement learning from the perspective of optimization and control with a focus on continuous control applications. It surveys the general formulation, terminology, and typical experimental implementations of reinforcement learning and reviews competing solution paradigms. In order to compa…
Certainty equivalent controllers perform nearly optimally in LQ control problems with unknown dynamics.
problem Optimizing control in systems with unknown transition dynamics.
method Analysis of certainty equivalent controllers and comparison to optimal LQ controllers, using perturbation bounds for discrete Riccati equations.
result Sub-optimality gap scales as the square of the parameter error, improving upon previous results.
This work establishes safe reinforcement learning for LQR with nonlinear baselines.
problem Safe reinforcement learning in LQR with unknown dynamics and safety constraints.
method General framework for nonlinear baselines, focusing on 1D spaces.
result Achieves optimal regret bounds for constrained reinforcement learning.
Algorithm minimizes control regret for non-stationary LQR systems.
problem Control of non-stationary LQR systems with unknown dynamics.
method Adaptive non-stationarity detection and OLS estimator with small bias.
result Achieves optimal dynamic regret of $ ilde{\mathcal{O}}\left(V_T^{2/5}T^{3/5}
ight)$ .
RichID learns optimal control policies from nonlinear observations.
problem Continuous control with unknown nonlinearity in system model.
method RichID algorithm using a least-squares regression oracle.
result First provable sample complexity guarantee for continuous control.
Perceptor Gradients learns symbolic representations from raw data.
problem Learning transferable symbolic representations from raw data.
method Decomposes policy into perceptor network and task encoding program.
result Efficiently learns symbolic representations for control tasks.
Study task-guided exploration in linear dynamical systems, improving sample complexity.
problem Efficiently learning about an environment to complete a specific task.
method Proposed a computationally efficient experiment-design based exploration algorithm.
result Optimally explores the environment, collecting precise information needed to complete the task.
Study shows a linear quadratic regulator's imitation learning converges globally.
problem Global convergence of imitation learning for linear quadratic regulators.
method Analyzed alternating gradient algorithm and established Q-linear rate of convergence.
result Established a unique saddle point for globally optimal policy and reward function.
Survey of theoretical foundations for policy optimization in control.
problem Understanding the theoretical properties of gradient-based methods in control and reinforcement learning.
method Interdisciplinary review of optimization landscape, convergence, and sample complexity for various control problems.
result Recent theoretical results on stability and robustness in learning-based control.
Optimal control strategy uses random noise to adaptively control systems with unknown parameters.
problem Online adaptive control of linear quadratic regulator with unknown system parameters.
method Certainty equivalent control with exploratory random noise, refined estimates of system matrices.
result Achieves optimal regret scaling as Θ(√(d_u^2 d_x T)) with self-bounding ODE method.
Safe control of systems with unknown dynamics using persistent excitation.
problem Tension between safety and exploration in data-driven control.
method System identification through persistent excitation, robust constraint satisfaction, and synthesis of feedback controllers.
result Non-asymptotic guarantees on estimation and controller performance.
Survey explores geometric aspects of policy optimization in control systems.
problem Understanding the geometric relationships between control design and optimization.
method Geometric perspective on policy optimization, focusing on parameterization and topology.
result Implications of policy geometry on stability and performance of local search algorithms.
Paper uses Koopman operator and Nyström method for efficient nonlinear control.
problem Control of nonlinear dynamical systems.
method Combines Koopman operator framework with Nyström approximation for kernel methods.
result Theoretical guarantees on the convergence rates of the approximated Riccati operator and regulator objective.
Efficient algorithm reduces control system learning regret to sqrt(T).
problem Learning Linear-Quadratic Regulators with unknown dynamics efficiently.
method First computationally-efficient algorithm with sqrt(T) regret.
result Resolves open question on control system learning.
New robust control method for uncertain systems using bootstrapped noise.
problem Designing controllers robust to model uncertainties in finite data.
method Least-squares model estimator, bootstrap resampling, multiplicative noise LQR.
result Significantly outperforms certainty equivalent controllers in numerical tests.
Paper solves time-inconsistent control problems with BSDEs.
problem Time-inconsistent stochastic control in continuous time.
method Probabilistic representation via BSDEs.
result Equilibrium value function resolved for inconsistent cases.
Policy gradient converges to globally optimal policy in nearly linear-quadratic systems.
problem Finding optimal policies in nonlinear control systems with partial information.
method Policy gradient algorithm designed for nearly linear-quadratic regulators with small Lipschitz nonlinear components.
result Policy gradient algorithm converges to globally optimal policy with linear rate.
Learning to control linear systems is statistically hard, especially for underactuated systems.
problem Statistical difficulty of learning to control linear systems, especially underactuated ones.
method Utilized minimax lower bounds and structural assumptions to prove learning complexity can be exponential.
result Learning complexity can be at most exponential with the controllability index of the system.
Algorithm improves continuous control with context, achieving provable efficiency.
problem Building an agent that generalizes to unseen environments in continuous control.
method Uses upper confidence bound (UCB) to refine decoder estimates and balance exploration-exploitation.
result Proves $\widetilde{O}\left(\sqrt{T}
ight)$ regret bound and ε ε ε -suboptimal policy transfer. We show LLMs can be locally linear, enabling better control of activations.
problem Suboptimal control of LLM activations during generation.
method Model LLM inference as a linear dynamical system, compute feedback controllers using Jacobians, and adapt classical control theory.
result Robust, fine-grained control of LLM activations across models and tasks.
The main challenge for adaptive regulation of linear-quadratic systems is the trade-off between identification and control. An adaptive policy needs to address both the estimation of unknown dynamics parameters (exploration), as well as the regulation of the underlying system (exploitation). To this end, optimism-based…
Reinforcement learning for optimizing retirement plans and target dated funds.
problem Optimizing financial goals through periodic investments and withdrawals.
method G-Learner and GIRL algorithms for goal-based wealth management.
result G-Learner provides a computationally tractable solution for wealth management tasks.
New algorithms achieve logarithmic regret in learning linear quadratic control systems.
problem Learning in Linear Quadratic Control systems with unknown parameters.
method Efficient algorithms for two scenarios: unknown A A A or B B B with certain conditions. result Regret scales logarithmically with the number of steps, not square root.
Study on variance of policy gradient in simple RL environments.
problem Understanding variance of policy gradient estimators in continuous RL.
method Analyzes REINFORCE estimator in linear-quadratic environments with Gaussian noise.
result Derives and validates bounds on estimator variance empirically.
Direct policy gradient methods for reinforcement learning and continuous control problems are a popular approach for a variety of reasons: 1) they are easy to implement without explicit knowledge of the underlying model 2) they are an "end-to-end" approach, directly optimizing the performance metric of interest 3) they…
This paper addresses the optimal control problem known as the Linear Quadratic Regulator in the case when the dynamics are unknown. We propose a multi-stage procedure, called Coarse-ID control, that estimates a model from a few experimental trials, estimates the error in that model with respect to the truth, and then d…
Modeling pollution from competing firms using mean-field games.
problem Pollution regulation of competitive firms producing similar goods.
method Developed a mean-field game model with cap-and-trade regulation.
result Explicit solutions found through Riccati differential equations.