New algorithm for context bandits with continuous actions.
problem Efficient decision-making with unknown action structures.
method Reduction-style algorithm combining supervised learning.
result Proven to work in general and validated with experiments.
Most existing deep reinforcement learning (DRL) frameworks consider either discrete action space or continuous action space solely. Motivated by applications in computer games, we consider the scenario with discrete-continuous hybrid action space. To handle hybrid action space, previous works either approximate the hyb…
AQL uses amortized inference to handle high-dimensional action spaces in Q-learning.
problem Difficulty in maximizing over large action spaces in Q-learning.
method Replace expensive maximization over all actions with a maximization over a small subset sampled from a learned proposal distribution.
result AQL outperforms existing methods on continuous control tasks with up to 21 dimensional actions.
Study on 2-valued dynamics on complex plane, showing some dynamics can't be group actions.
problem Whether 2-valued dynamics can be defined by the action of a 2-valued group.
method Construction of examples of dynamics that are or are not group actions.
result Some 2-valued dynamics on complex plane cannot be defined by the action of a 2-valued group.
Any continuous action of SL(n,Z), where n > 2, on a r-dimensional mod 2 homology sphere factors through a finite group action if r < n - 1. In particular, any continuous action of SL(n+2,Z) on the n-dimensional sphere factors through a finite group action.
A core novelty of Alpha Zero is the interleaving of tree search and deep learning, which has proven very successful in board games like Chess, Shogi and Go. These games have a discrete action space. However, many real-world reinforcement learning domains have continuous action spaces, for example in robotic control, na…
Smooth approximations for continuous functions on orbit spaces.
problem Approximating continuous functions on orbit spaces.
method Study of subcartesian spaces and proper Lie group actions.
result Continuous functions can be approximated by smooth functions.
Hybrid SAC improves RL for video games with discrete, continuous actions.
problem Improving RL performance in video games with practical constraints.
method Extension of Soft Actor-Critic (SAC) for handling discrete, continuous, and parameterized actions.
result Hybrid SAC successfully solves a high-speed driving task and is competitive on parameterized actions benchmarks.
The paper tackles counterfactual learning for stochastic policies with continuous actions.
problem Learning stochastic policies with continuous actions from logged data.
method Introduces a joint kernel embedding of contexts and actions to model continuous actions, and uses proximal point algorithms and smooth estimators for optimization.
result Demonstrates the benefits of using proximal point algorithms and smooth estimators for counterfactual learning.
Novel framework proves fast RL convergence in continuous spaces.
problem Analyzing stability in continuous state-action RL.
method Introduces a novel framework to analyze stability properties of RL.
result Highlights two key stability properties and demonstrates their satisfaction in RL.
Let M1 and M2 be two n-dimensional smooth manifolds with boundary. Suppose we glue M1 and M2 along some boundary components (which are, therefore, diffeomorphic). Call the result N. If we have a group G acting continuously on M1, and also acting continuously on M2, such that the actions are comp…
Continuous reinforcement learning such as DDPG and A3C are widely used in robot control and autonomous driving. However, both methods have theoretical weaknesses. While DDPG cannot control noises in the control process, A3C does not satisfy the continuity conditions under the Gaussian policy. To address these concerns,…
New method extends low-rank MDPs to continuous action spaces.
problem Limited applicability of current low-rank MDP methods to continuous action spaces.
method Extending FLAMBE algorithm to continuous action spaces with Hölder smoothness conditions.
result Similar PAC bound achieved for continuous actions with polynomial dependence on smoothness order.
Value-based reinforcement learning (RL) methods like Q-learning have shown success in a variety of domains. One challenge in applying Q-learning to continuous-action RL problems, however, is the continuous action maximization (max-Q) required for optimal Bellman backup. In this work, we develop CAQL, a (class of) algor…
Efficient algorithms for contextual bandits with smooth regret in continuous action spaces.
problem Efficient learning in large or continuous action spaces.
method Smooth regret notion and efficient algorithms for general function approximation.
result Statistically and computationally efficient algorithms for contextual bandits with smooth regret.
Proposes hybrid reinforcement learning for both discrete and continuous control problems.
problem Real-world control problems involving both discrete and continuous decision variables.
method Solves hybrid problems by optimizing for discrete and continuous actions simultaneously.
result Efficiently solves hybrid reinforcement learning problems and improves upon expert heuristics.
Temporal-difference (TD) networks are a class of predictive state representations that use well-established TD methods to learn models of partially observable dynamical systems. Previous research with TD networks has dealt only with dynamical systems with finite sets of observations and actions. We present an algorithm…
Proper actions on bornological spaces are characterized with compatible coarse structures.
problem Characterizing proper actions on bornological spaces.
method Proving the existence of compatible coarse structures for proper actions.
result Bornological spaces admit compatible coarse structures for proper actions.
Hybrid RL method optimizes trading by balancing continuous and discrete actions.
problem Optimal execution in algorithmic trading with continuous-discrete action space.
method Combines continuous and discrete RL agents for better trading decisions.
result Significantly outperforms existing methods in trading efficiency and stability.
Hybrid Policy Optimization tackles reinforcement learning in hybrid spaces, improving performance over PPO.
problem Credit assignment issues and biased gradients in hybrid discrete-continuous action spaces.
method Mixed gradient estimator combining pathwise and score-function gradients, reformulating problems in hybrid form.
result HPO substantially outperforms PPO on inventory control and switched systems, with performance gaps increasing with continuous action dimension.
Paper tackles RL with continuous actions and unmeasured confounders.
problem Offline policy learning with continuous actions and unmeasured confounders.
method Developed a novel identification result and a minimax estimator for nonparametric policy value estimation.
result Introduced a policy-gradient-based algorithm to identify the optimal policy.
We show that every continuous action of a finite group on a smooth three-manifold is a uniform limit of smooth actions.
Many continuous control tasks have bounded action spaces. When policy gradient methods are applied to such tasks, out-of-bound actions need to be clipped before execution, while policies are usually optimized as if the actions are not clipped. We propose a policy gradient estimator that exploits the knowledge of action…
The paper explores properties of continuous actions on manifolds, proving bounds on subgroup size and fixed points.
problem Properties of continuous finite group actions on topological manifolds.
method Analyzes properties including Jordan property and almost fixed point property, proving bounds on subgroup size.
result Existence of a constant C such that for any continuous action of a finite group G on a manifold X, there is a subgroup H with [G:H] ≤ C and a fixed point.
New proof shows path-connectedness of actions on intervals and circles.
problem Path-connectedness of C1+ac actions of Zd. method New proof using C1 diffeomorphisms with absolutely continuous derivative. result Path-connectedness of the space of actions.
Action chunking and data exploration improve behavior cloning in robotics.
problem Exponential errors in learning from demonstrations for continuous control tasks.
method Action chunking and exploratory data collection.
result Control-theoretic stability is key to improving imitation learning.
The paper proposes a method to learn continuous-action graphical games from perturbed equilibria.
problem Learning the exact structure of continuous-action graphical games from limited data.
method A ℓ12− block regularized method to recover the graphical game structure. result The method recovers the exact structure of the graphical game under certain conditions.
The paper proposes a method to learn the structure of continuous-action games with non-parametric utilities using a limited number of samples.
problem Learning the exact structure of continuous-action games with non-parametric utility functions.
method An ℓ1 regularized method that encourages sparsity of the Fourier transform coefficients of the utility functions, accessed via a few Nash equilibria and their noisy utilities. result The method recovers the exact structure of the utility functions and the game structure with provable theoretical guarantees.
A hybrid model combines Q-learning and PID controller for continuous vehicle control.
problem Learning unsatisfactory results with discrete action space in autonomous driving.
method Combining Q-learning and PID controller, using Quadratic Q-function approximation and action network.
result Autonomous vehicle successfully learns smooth and efficient driving behavior.
Recent work has shown that reinforcement learning (RL) is a promising approach to control dynamical systems described by partial differential equations (PDE). This paper shows how to use RL to tackle more general PDE control problems that have continuous high-dimensional action spaces with spatial relationship among ac…
RANDPOL uses randomized networks for efficient reinforcement learning in continuous state and action MDPs.
problem Efficient reinforcement learning in environments with continuous state and action spaces.
method RANDPOL uses randomized function approximation to represent policy and value functions, providing finite time guarantees and improved numerical performance.
result RANDPOL achieves better numerical performance and provides finite time guarantees compared to deep neural network based algorithms.
The paper extends Thompson Sampling to infinite action spaces using information theory.
problem Addressing the limitation of finite action spaces in Thompson Sampling.
method Information-theoretic analysis, extending rate-distortion theory to infinite action spaces.
result Derives a near-optimal regret bound for bandits with infinite and continuous action spaces.
New framework discovers non-affine continuous symmetries in neural networks.
problem Lack of efficient methods for detecting non-affine continuous symmetries in neural networks.
method Computational framework for discovering infinitesimal generators of multi-parameter group actions.
result Framework can discover non-affine continuous symmetries in neural networks.
In this paper we study perturbations of constant cocycles for actions of higher rank semi-simple algebraic groups and their lattices. Roughly speaking, for ergodic actions, Zimmer's cocycle superrigidity theorems implies that the perturbed cocycle is measurably conjugate to a constant cocycle modulo a compact valued co…
Models analyze strategic risk-taking in continuous action games.
problem Strategic risk-taking dynamics in continuous action games.
method Normal form game, multi-player scenarios, regret minimization algorithms, numerical algorithm for calculation.
result Nash equilibrium also serves as a correlated equilibrium in continuous games.
Deep RBVFs improve continuous control in RL.
problem Challenges in finding optimal actions for continuous actions in RL.
method Introduced deep radial-basis value functions (RBVFs) for continuous control.
result RBF-DQN significantly outperforms value-function-only baselines and is competitive with actor-critic algorithms.
Develops a statistical learning framework for personalized asset allocation.
problem Continuous-action decision-making with a large number of characteristics.
method Discretization approach with generalized penalties for penalized regression.
result Improves financial well-being with individualized optimal asset allocation.
Proposes a value-based method for continuous control without an actor.
problem Computational infeasibility of evaluating Q-values in continuous action spaces.
method Structurally maximizable Q-functions, actor-free approach.
result Performance and sample efficiency comparable to actor-critic methods.
We study contextual bandit learning with an abstract policy class and continuous action space. We obtain two qualitatively different regret bounds: one competes with a smoothed version of the policy class under no continuity assumptions, while the other requires standard Lipschitz assumptions. Both bounds exhibit data-…
Develops a new RL algorithm for medical treatment regimes.
problem Optimal dose determination in continuous action environments.
method Quasi-optimal learning algorithm for near-optimal actions.
result Guaranteed convergence and effectiveness in real applications.
Safe imitation learning with a safety layer for flexible training.
problem Flexible yet safe imitation learning for complex tasks.
method Theory and modular method with a safety layer for continuous policy, adversarial training, and worst-case safety guarantees.
result Robustness advantage of safety layer during training compared to test time.
This paper proposes a novel deep reinforcement learning architecture that was inspired by previous tree structured architectures which were only useable in discrete action spaces. Policy Prediction Network offers a way to improve sample complexity and performance on continuous control problems in exchange for extra com…
Practical algorithm for contextual bandits with large action spaces.
problem Efficient algorithms for decision making in large, continuous action spaces.
method Uses computational oracles for supervised learning and optimization over the action space.
result Achieves sample complexity, runtime, and memory independent of the size of the action space.
Deep Reinforcement Learning (DRL) has been applied to address a variety of cooperative multi-agent problems with either discrete action spaces or continuous action spaces. However, to the best of our knowledge, no previous work has ever succeeded in applying DRL to multi-agent problems with discrete-continuous hybrid (…
In this article, we sketch an algorithm that extends the Q-learning algorithms to the continuous action space domain. Our method is based on the discretization of the action space. Despite the commonly used discretization methods, our method does not increase the discretized problem dimensionality exponentially. We wil…
Study a continuous-time PA problem with private effort and consumption decisions.
problem Continuous-time Principal-Agent problem with private information.
method Proposes a new sufficient condition for solving the agent's problem directly.
result Directly yields a solution to the agent's problem without verification.
It has long been assumed that high dimensional continuous control problems cannot be solved effectively by discretizing individual dimensions of the action space due to the exponentially large number of bins over which policies would have to be learned. In this paper, we draw inspiration from the recent success of sequ…
We introduce a framework for model learning and planning in stochastic domains with continuous state and action spaces and non-Gaussian transition models. It is efficient because (1) local models are estimated only when the planner requires them; (2) the planner focuses on the most relevant states to the current planni…