arXiv research

Simple stochastic Newton and cubic Newton methods with fast convergence.

Policy gradient converges to globally optimal policy in nearly linear-quadratic systems.

We show by counterexample that policy-gradient algorithms have no guarantees of even local convergence to Nash equilibria in continuous action and state space multi-agent settings. To do so, we analyze gradient-play in N-player general-sum linear quadratic games, a classic game setting which is recently emerging as a b…

Policy gradient methods find Nash equilibrium in noisy games.

We study the global convergence of policy optimization for finding the Nash equilibria (NE) in zero-sum linear quadratic (LQ) games. To this end, we first investigate the landscape of LQ games, viewing it as a nonconvex-nonconcave saddle-point problem in the policy space. Specifically, we show that despite its nonconve…

Global convergence proved for multi-agent LQRs with hierarchical actor-critic.

Study on PG learning for LQ MFC problems with common noise, proving convergence and sample complexity.

We study the global convergence of generative adversarial imitation learning for linear quadratic regulators, which is posed as minimax optimization. To address the challenges arising from non-convex-concave geometry, we analyze the alternating gradient algorithm and establish its Q-linear rate of convergence to a uniq…

RL solves discrete LQ control with Gaussian optimal policy.

Paper uses Koopman operator and Nyström method for efficient nonlinear control.

Policy gradient methods converge for LQR problems with noisy state dynamics.

Despite the empirical success of the actor-critic algorithm, its theoretical understanding lags behind. In a broader context, actor-critic can be viewed as an online alternating update algorithm for bilevel optimization, whose convergence is known to be fragile. To understand the instability of actor-critic, we focus o…

We study derivative-free methods for policy optimization over the class of linear policies. We focus on characterizing the convergence rate of these methods when applied to linear-quadratic systems, and study various settings of driving noise and reward feedback. We show that these methods provably converge to within a…

Study policy gradient for large-agent mean-field control and game in continuous time.

Solves steering problem with continuous time, Hilbert-Schmidt cost, and matrix ODEs.

Learning to make decisions from observed data in dynamic environments remains a problem of fundamental importance in a number of fields, from artificial intelligence and robotics, to medicine and finance. This paper concerns the problem of learning control policies for unknown linear dynamical systems so as to maximize…

Direct policy gradient methods for reinforcement learning and continuous control problems are a popular approach for a variety of reasons: 1) they are easy to implement without explicit knowledge of the underlying model 2) they are an "end-to-end" approach, directly optimizing the performance metric of interest 3) they…

Regularized least-squares approaches have been successfully applied to linear system identification. Recent approaches use quadratic penalty terms on the unknown impulse response defined by stable spline kernels, which control model space complexity by leveraging regularity and bounded-input bounded-output stability. T…

The paper studies optimal transport in linear quadratic systems and derives interpolation inequalities.

Survey of theoretical foundations for policy optimization in control.

Study optimizes investment strategies in markets with contagious price jumps.

Algorithm reduces regret in partially observable systems by learning dynamics and using optimistic control.

New algorithms achieve logarithmic regret in learning linear quadratic control systems.

We present the first computationally-efficient algorithm with $\widetilde O(\sqrt{T})$ regret for learning in Linear Quadratic Control systems with unknown dynamics. By that, we resolve an open question of Abbasi-Yadkori and Szepesvári (2011) and Dean, Mania, Matni, Recht, and Tu (2018).

COMRADE is a communication-efficient, Byzantine-resilient second-order optimization algorithm.

Logarithmic regret achieved in continuous-time linear-quadratic reinforcement learning.

LqgOpt learns optimal control in unknown LQG systems with minimal regret.

We study the performance of the certainty equivalent controller on Linear Quadratic (LQ) control problems with unknown transition dynamics. We show that for both the fully and partially observed settings, the sub-optimality gap between the cost incurred by playing the certainty equivalent controller on the true system …

Data processing inequalities link Fisher information to local differential privacy constraints.

We study the problem of adaptive control of a high dimensional linear quadratic (LQ) system. Previous work established the asymptotic convergence to an optimal controller for various adaptive control schemes. More recently, for the average cost LQ problem, a regret bound of ${O}(\sqrt{T})$ was shown, apart form logarit…

We formulate a general framework for competitive gradient-based learning that encompasses a wide breadth of multi-agent learning algorithms, and analyze the limiting behavior of competitive gradient-based learning algorithms using dynamical systems theory. For both general-sum and potential games, we characterize a non…

Random extrapolation speeds up coordinate descent for sparse and dense data.

Study shows certainty equivalent policy minimizes regret in continuous-time systems.

New algorithm learns LQR with $O(\sqrt{T})$ regret using Langevin dynamics and excitation.

Optimizes dividends with stability for risky businesses.

This paper establishes the existence of a unique nonnegative continuous viscosity solution to the HJB equation associated with a Markovian linear-quadratic control problems with singular terminal state constraint and possibly unbounded cost coefficients. The existence result is based on a novel comparison principle for…

We propose a randomized second-order method for optimization known as the Newton Sketch: it is based on performing an approximate Newton step using a randomly projected or sub-sampled Hessian. For self-concordant functions, we prove that the algorithm has super-linear convergence with exponentially high probability, wi…

New algorithms improve blind source separation for linear-quadratic mixtures.

Novel approach to Nash equilibrium in mean-field stochastic games with operator resolvents.

Develops a control framework for systemic risk under uncertainty.

We study the constrained linear quadratic regulator with unknown dynamics, addressing the tension between safety and exploration in data-driven control techniques. We present a framework which allows for system identification through persistent excitation, while maintaining safety by guaranteeing the satisfaction of st…

We consider a general time-inconsistent stochastic linear-quadratic differential game. The time-inconsistency arises from the presence of quadratic terms of the expected state as well as state-dependent term in the objective functionals. We define an equilibrium strategy, which is different from the classical one, and …

New theory extends LQ control to non-exponential discount scenarios.

Study optimizes resource allocation in noisy systems for better control.

Efficiently solves exploration-exploitation in LQR using Lagrangian relaxation.

Paper solves time-inconsistent control problems with BSDEs.

Survey explores geometric aspects of policy optimization in control systems.

In optimization, the negative gradient of a function denotes the direction of steepest descent. Furthermore, traveling in any direction orthogonal to the gradient maintains the value of the function. In this work, we show that these orthogonal directions that are ignored by gradient descent can be critical in equilibri…