Algorithm tackles constrained reinforcement learning with concave-convex and knapsack constraints.
problem Constrained episodic reinforcement learning with concave rewards and convex constraints.
method Modular analysis with strong theoretical guarantees for concave-convex and knapsack settings.
result Significantly outperforms existing approaches in constrained episodic environments.
Develops new reinforcement learning methods for complex constrained decision-making problems.
problem Complex constrained decision-making problems with a continuum of constraints.
method Proposes semi-infinitely constrained Markov decision processes (SICMDPs) and two reinforcement learning algorithms: SI-CRL and SI-CPO.
result Demonstrates the effectiveness of SI-CRL and SI-CPO in solving complex sequential decision-making tasks.
Safe RL in linear systems achieves T \sqrt{T} T -regret.
problem Efficiently learning in safety-constrained online reinforcement learning.
method Study of linear quadratic regulator with safety constraints.
result First safe algorithm with i l d e O T ( T ) ilde{O}_T(\sqrt{T}) i l d e O T ( T ) -regret. Novel methods generate diverse policies in reinforcement learning.
problem Generating diverse policies in reinforcement learning.
method Constrained optimization perspective, introducing new metrics, and novel policy generation methods.
result Improved novelty and performance of generated policies.
Paper uses RL to solve constrained combinatorial optimization problems.
problem Constrained combinatorial optimization problems.
method Extending NCO theory to CMDPs, using RL with penalty signals.
result Proposes a superior method for solving constrained problems.
Eikonal-Constrained QRL improves goal-reaching in reinforcement learning.
problem Reward design and out-of-distribution generalization in reinforcement learning.
method Eikonal-Constrained Quasimetric Reinforcement Learning (Eik-QRL) using the Eikonal PDE.
result Eik-QRL achieves state-of-the-art performance in offline goal-conditioned navigation and manipulation tasks.
Oracle-efficient algorithm for offline RL with partial data coverage.
problem Offline reinforcement learning with partial data coverage and constraints.
method PDOCRL, a primal-dual algorithm with decomposed linear-programming formulation.
result Near-optimal, near-feasible policy with \(\widetilde{\mathcal O}(ε^{-2})\) sample guarantee.
Paper tackles efficient navigation in constrained environments using supervised and reinforcement learning.
problem Building learning agents that efficiently navigate in obstacle-cluttered environments.
method Synergistic use of supervised learning for path prediction and reinforcement learning for path following.
result Proposed method achieves good generalization and faster learning compared to existing work.
Algorithm mitigates performance loss in constrained reinforcement learning with model misspecification.
problem Performance loss in reinforcement learning policies due to model misspecification in constrained control systems.
method Proposes an algorithm to handle constrained model misspecification in continuous control systems.
result Algorithm successfully mitigates performance loss in real-world reinforcement learning tasks.
Proposes Constrained Q-learning for reinforcement learning with constraints.
problem Optimizing multiple objectives while adhering to constraints in reinforcement learning.
method Directly restricts the action space in Q-update to learn optimal Q-function for constrained MDP.
result Improves safety and optimality in high-level decision making for autonomous driving.
State-constrained offline RL expands RL's learning scope.
problem Batch-constrained offline RL limits policies to seen actions.
method Introduces state-constrained offline RL focusing on state distribution.
result Policy can take high-quality out-of-distribution actions.
Many practical applications of reinforcement learning constrain agents to learn from a fixed batch of data which has already been gathered, without offering further possibility for data collection. In this paper, we demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learn…
Paper proposes risk-averse reinforcement learning algorithms.
problem Managing model uncertainty in reinforcement learning.
method Entropic risk constrained policy gradient and actor-critic algorithms.
result Demonstrates usefulness of risk-averse algorithms on various domains.
Algorithm optimizes constrained reinforcement learning with dual variables.
problem Minimizing convex functional subject to convex constraint in large state spaces.
method VPDPO algorithm using Lagrangian and Fenchel duality.
result Achieves sublinear regret and constraint violation, globally optimal policy.
Novel evolutionary strategy solves stochastic constrained optimization problems.
problem Optimizing objective functions with stochastic constraints in reinforcement learning.
method Design of a novel optimization algorithm with a sufficient decrease mechanism for stochastic constrained problems.
result Demonstrated convergence of the algorithm on control tasks and constrained optimization problems.
Study optimality in safety-constrained Markov decision processes using asynchronous value iteration and modified Q-learning.
problem Optimality in safety-constrained Markov decision processes with multichain structure.
method Formulated as a zero-sum game, constructed asynchronous value iteration scheme and modified Q-learning algorithm.
result Resolved Bellman's principle of optimality for multichain Markov decision processes and provided learning algorithms.
We study the safe reinforcement learning problem with nonlinear function approximation, where policy optimization is formulated as a constrained optimization problem with both the objective and the constraint being nonconvex functions. For such a problem, we construct a sequence of surrogate convex constrained optimiza…
Solves online 3D bin packing with deep reinforcement learning under constraints.
problem Challenges of packing items immediately without information and constraints.
method Constrained deep reinforcement learning (DRL) with feasibility predictor.
result Significantly outperforms state-of-the-art methods in online 3D bin packing.
PDCA algorithm learns policies for RL with constraints using a primal-dual approach.
problem Offline constrained reinforcement learning with general function approximation.
method Primal-Dual-Critic Algorithm (PDCA) using a primal-dual approach.
result PDCA finds a near saddle point of the Lagrangian, nearly optimal for constrained RL.
Opportunistic spectrum access is one of the emerging techniques for maximizing throughput in congested bands and is enabled by predicting idle slots in spectrum. We propose a kernel-based reinforcement learning approach coupled with a novel budget-constrained sparsification technique that efficiently captures the envir…
VaR-CPO optimizes VaR-constrained RL problems with conservative policy updates.
problem Optimizing VaR-constrained reinforcement learning problems.
method Combines Cantelli's inequality and trust-region framework for efficient and conservative optimization.
result Achieves zero constraint violations during training in feasible environments.
KG-A2C agent learns natural language IF games by reasoning and constraining action spaces.
problem Challenges of natural language understanding, partial observability, and combinatorially large action spaces in IF games.
method Builds a dynamic knowledge graph while exploring, constraining actions using templates.
result Outperforms current IF agents across various games with larger action spaces.
ClauseLens uses reinforcement learning to price reinsurance treaties transparently and auditably.
problem Opaque and difficult-to-audit reinsurance treaty pricing practices.
method ClauseLens models treaty pricing as a Risk-Aware Constrained Markov Decision Process (RA-CMDP), incorporating legal clauses and generating interpretable explanations.
result ClauseLens reduces solvency violations and improves tail-risk performance, achieving 88.2% accuracy in clause-grounded explanations.
KCRL learns stable policies for nonlinear systems with formal guarantees.
problem Lack of stabilization guarantees in RL methods for safety-critical systems.
method KCRL uses Krasovskii's Lyapunov functions as a stability constraint and a primal-dual approach to learn stabilizing policies.
result KCRL guarantees learning a stabilizing policy in a finite number of interactions.
Study uses DRL with Lagrangian relaxation to solve temporal control tasks with STL constraints.
problem Optimal control problems with temporal logic constraints.
method Extended CMDP formulation, Lagrangian relaxation, two-phase constrained DRL algorithm.
result Demonstrated learning performance of the proposed algorithm through simulations.
New algorithm learns safe policies in unknown environments.
problem Learning safe policies in unknown, potentially unsafe environments.
method C-UCRL: Upper Confidence Reinforcement Learning for constrained MDPs.
result Achieves sub-linear regret while satisfying constraints.
KL-constrained API shows optimization issues and improved with regularization.
problem Optimization issues in KL-constrained API algorithms.
method Comparison of KL divergence as a constraint vs. regularizer, empirical evaluation.
result KL-constrained API is not guaranteed to converge and incurs linear regret.
This work establishes safe reinforcement learning for LQR with nonlinear baselines.
problem Safe reinforcement learning in LQR with unknown dynamics and safety constraints.
method General framework for nonlinear baselines, focusing on 1D spaces.
result Achieves optimal regret bounds for constrained reinforcement learning.
FOCOPS optimizes agent's behavior while adhering to constraints.
problem Optimizing agent's behavior while respecting safety constraints.
method FOCOPS solves a constrained optimization problem in policy space, then projects the solution back into the parametric space.
result FOCOPS achieves better performance on constrained robotics tasks.
Safe RL for autonomous vehicles using PCPO with trust regions and parallel learners.
problem Unexplainable behaviours and lack of safety guarantees in RL for real vehicles.
method PCPO framework with trust regions and parallel learners.
result Safe learning confirmed for autonomous vehicles with fast convergence.
Survey of mathematical foundations for reinforcement learning.
problem Design and analysis of modern reinforcement learning algorithms.
method Organizes mathematical structures from probability, optimization, and operator theory.
result Unified mathematical entry point for researchers in various fields.
We propose a novel training algorithm for reinforcement learning which combines the strength of deep Q-learning with a constrained optimization approach to tighten optimality and encourage faster reward propagation. Our novel technique makes deep reinforcement learning more practical by drastically reducing the trainin…
New algorithm for reinforcement learning in uncertain environments with unknown thresholds.
problem Safety in reinforcement learning in unknown and uncertain environments.
method Growing-Window estimator sampling and Stochastic Pessimistic-Optimistic Thresholding (SPOT) algorithm.
result Achieves sublinear regret and constraint violation of i l d e O ( T ) ilde{\mathcal{O}}(\sqrt{T}) i l d e O ( T ) . Canary optimizes VaR-constrained RL problems with a conservative bound using Cantelli's inequality.
problem Optimizing reinforcement learning policies under VaR constraints in dense cost regimes.
method Employing Cantelli's inequality to create a conservative and smooth bound on VaR constraints based on moments of cost returns. Extending trust-region framework for worst-case bounds on policy improvement and constraint violation.
result Canary reliably satisfies VaR constraints with fewest violations and earliest permanent satisfaction, while maintaining reward competitiveness.
FMDP-BF algorithm improves RL in factored MDPs with exponential regret reduction.
problem Efficient reinforcement learning in factored MDPs with constrained RL.
method FMDP-BF algorithm leveraging factorization structure of FMDPs.
result FMDP-BF's regret is exponentially smaller than optimal algorithms for non-factored MDPs.
Paper robustifies reinforcement learning with risk-averse methods.
problem Making predictions robust to changes in system dynamics or rewards.
method Approximates Robust Reinforcement Learning using Φ Φ Φ -divergence and Risk-Averse formulation. result Classical Reinforcement Learning can be robustified using standard deviation penalization.
Study examines sample complexity for RL with safety constraints.
problem Ensuring safety in RL for systems with constraints.
method Analyzes two RL approaches: generative and online.
result Sample complexity increased by a logarithmic factor in constraints.
Study proposes adaptive RL for dynamic portfolio optimization.
problem Traditional portfolio optimization models fail to adapt to regime shifts.
method Regime-aware reinforcement learning framework with hybrid observations and constrained reward functions.
result Transformer PPO achieves highest risk-adjusted returns, while LSTM variants offer a good balance.
New algorithm reduces sample complexity for safe reinforcement learning.
problem Safe reinforcement learning in constrained MDPs with performance and safety constraints.
method Model-based primal-dual algorithm balancing regret and bounded constraint violations.
result Proves near-optimal policies with bounded violations or zero violations in CMDPs.
A new RL method handles uncertainty and constraints in real-time optimization.
problem Real-time optimization under process uncertainty and constraints.
method Chance-constrained reinforcement learning to handle probabilistic state constraints.
result Satisfies process constraints with high probability in real-time.
A novel approach for safe offline RL using latent safety constraints.
problem Balancing safety constraints and reward maximization in offline RL.
method Conditional Variational Autoencoders for latent safety modeling, Constrained Reward-Return Maximization.
result Our approach maintains safety compliance while optimizing rewards, outperforming existing methods.
Study uses RL to optimize investment with financial constraints, showing exploration benefits.
problem Optimal investment with financial constraints in continuous time.
method Reinforcement learning framework, focusing on Gaussian and truncated Gaussian distributions.
result Exploration leads to more dispersed wealth distribution with heavier tails, especially with smaller exploration parameters.
Improved RL algorithm with linear MDPs for offline learning with partial data coverage.
problem Efficient offline RL with linear MDPs under partial data coverage.
method Primal-dual algorithm with O ( ε − 2 ) O(ε^{-2}) O ( ε − 2 ) sample complexity. result First computationally efficient algorithm with O ( ε − 2 ) O(ε^{-2}) O ( ε − 2 ) sample complexity for offline RL with linear MDPs under partial data coverage. A novel method for efficient CDRL over wireless networks.
problem Challenges in collaborative deep reinforcement learning over wireless networks.
method Semantic-aware heterogeneous federated deep reinforcement learning (HFDRL) algorithm.
result Superior performance compared to state-of-the-art baselines.
New algorithm for risk-sensitive reinforcement learning with natural policy gradients.
problem Risk-sensitive reinforcement learning with downside risk constraints.
method Introduce a new Bellman equation to estimate the lower partial moment of returns, use natural policy gradients, and extend Reward Constrained Policy Optimization.
result Sample-efficient estimation of partial moments and effective risk-sensitive control.
New RL approach tackles constrained Markov decision processes.
problem Applying RL to physical systems with safety constraints.
method Formulated as a Constrained Markov Decision Process (CMDP), introduced a safe policy improvement method.
result Agent learns to maximize returns while satisfying constraints.
FISAR uses neural networks to optimize safe reinforcement learning with forward-invariant constraints.
problem Safe reinforcement learning with constraints in safety-critical environments.
method Imposing linear constraints on policy parameters' updating dynamics, using a DNN-based optimizer to satisfy these constraints.
result The policy decreases constraint violation and maximizes cumulative reward monotonically.
Solving tasks in Reinforcement Learning is no easy feat. As the goal of the agent is to maximize the accumulated reward, it often learns to exploit loopholes and misspecifications in the reward signal resulting in unwanted behavior. While constraints may solve this issue, there is no closed form solution for general co…