New algorithm stabilizes RL policy learning through divergence regularization.
problem Stabilize policy learning and improve performance in RL.
method Proximity term constraining discounted state-action visitation distributions to be close to each other.
result Proposed algorithm improves stability and final performance in RL tasks.
The paper examines how macroeconomic control tools lost effectiveness, leading to a 'dark ages' period.
problem Loss of effectiveness of control tools in macroeconomic stabilization policy.
method Historical analysis of macroeconomic stabilization policy from 1948 to 1993.
result The overstatement of the Lucas critique and Kydland and Prescott's time-inconsistency led to a period of ineffective stabilization policy.
Improves reinforcement learning stability and efficiency.
problem Combining stability and efficiency in reinforcement learning.
method Combines on-policy stability with off-policy sample reuse.
result Demonstrates improved performance in both theory and practice.
KCRL learns stable policies for nonlinear systems with formal guarantees.
problem Lack of stabilization guarantees in RL methods for safety-critical systems.
method KCRL uses Krasovskii's Lyapunov functions as a stability constraint and a primal-dual approach to learn stabilizing policies.
result KCRL guarantees learning a stabilizing policy in a finite number of interactions.
Algorithmic stablecoins optimize monetary policy to balance price stability.
problem Persistent inflation from centralized monetary policy.
method Propose and study a rule-based monetary policy model for algorithmic stablecoins.
result Optimal trade-off between price stability and supply stability.
Survey explores geometric aspects of policy optimization in control systems.
problem Understanding the geometric relationships between control design and optimization.
method Geometric perspective on policy optimization, focusing on parameterization and topology.
result Implications of policy geometry on stability and performance of local search algorithms.
The paper presents a model-free method for stabilizing unknown control systems.
problem Stabilizing unknown control systems in engineering.
method Solving discounted LQR problems with increasing discount factors.
result The method efficiently recovers a stabilizing controller for linear and smooth nonlinear systems.
Adapts model-based advice to stabilize black-box policies for nonlinear control.
problem Stabilizing machine-learned policies for nonlinear control with limited model information.
method Proposes an adaptive λ-confident policy to combine black-box and model-based advice. result Proves the stability of the adaptive λ-confident policy and its competitive ratio. Develops ODRPO to improve RL algorithms with better performance and stability.
problem RL algorithms converge to sub-optimal solutions due to limited policy representation.
method Integrates DRO approach to solve trust region constrained optimization problem without parameterizing policies.
result Achieves globally optimal policy update and higher sample efficiency.
Adaptive optimal control using value iteration (VI) initiated from a stabilizing policy is theoretically analyzed in various aspects including the continuity of the result, the stability of the system operated using any single/constant resulting control policy, the stability of the system operated using the evolving/ti…
This paper investigates a type of instability that is linked to the greedy policy improvement in approximated reinforcement learning. We show empirically that non-deterministic policy improvement can stabilize methods like LSPI by controlling the improvements' stochasticity. Additionally we show that a suitable represe…
Deep Reinforcement Learning (DRL) algorithms for continuous action spaces are known to be brittle toward hyperparameters as well as \cut{being}sample inefficient. Soft Actor Critic (SAC) proposes an off-policy deep actor critic algorithm within the maximum entropy RL framework which offers greater stability and empiric…
New algorithms improve policy evaluation in reinforcement learning.
problem Off-policy stability and on-policy efficiency issues in policy evaluation.
method Introduced novel algorithms using oblique projection method.
result Demonstrated both off-policy stability and on-policy efficiency.
Embedding RL policies in RKHS for robustness and theoretical guarantees.
problem Stability and theoretical guarantees in RL policy representation.
method Low-dimensional embedding of RL policies in RKHS.
result Embedded policies maintain high return with strong theoretical guarantees.
New RL policy for unbounded state space with stability guarantee.
problem Traditional RL methods fail for unbounded state space.
method Proposes stability as performance metric, uses Sparse-Sampling-based Monte Carlo Oracle.
result Proposed policy ensures state dynamics remain bounded with high probability.
Stabilizes policy optimization with off-policy data using divergence augmentation.
problem Premature convergence and instability in policy optimization with off-policy data.
method Incorporates Bregman divergence between behavior and current policies to ensure safe policy updates.
result Empirically shows better performance in data-scarce scenarios compared to other algorithms.
The paper optimizes policies constrained to Schur stabilizing controllers using a Newton-type algorithm.
problem Optimizing policies under linear constraints in control systems.
method Newton-type algorithm on a manifold of Schur stabilizing controllers with a Riemannian metric.
result Local convergence guarantees for the Newton-type algorithm without relying on exponential mapping or retractions.
Adaptive optimal control using value iteration initiated from a stabilizing control policy is theoretically analyzed in terms of stability of the system during the learning stage without ignoring the effects of approximation errors. This analysis includes the system operated using any single/constant resulting control …
Financial system being the place of metting capital flows (equality between saving and investment), a volatility of capital flows can destroy the robustness and good working of financial system, it means subvert financial stability. The same a weak financial system, few regulated and bad manage can exacerbate volatilit…
Value aggregation is a general framework for solving imitation learning problems. Based on the idea of data aggregation, it generates a policy sequence by iteratively interleaving policy optimization and evaluation in an online learning setting. While the existence of a good policy in the policy sequence can be guarant…
The paper shows how to stabilize off-policy reinforcement learning using specific state representations.
problem Stability issues in reinforcement learning with function approximation and off-policy learning.
method Formal analysis of representation learning schemes based on the transition matrix of a policy.
result Schur and orthogonal bases of the Krylov subspace provide stable representations for TD learning.
Logit dynamics formula reveals self-regulation in softmax policy gradient methods.
problem Understanding the stability and convergence of softmax policy gradient methods.
method Deriving the exact formula for the L2 norm of the logit update vector.
result Logit update magnitudes are modulated by action probability and policy concentration.
New method stabilizes FQE by reweighting Bellman targets.
problem Stability guarantees for FQE often rely on Bellman completeness, which can fail with function approximation.
method Proposes stationary-weighted FQE, reweighting Bellman targets by stationary target-to-behavior density ratio.
result Proves finite-sample linear convergence to stationary projected Bellman fixed point without Bellman completeness.
ADAC uses analogous policies to improve RL exploration without sacrificing stability.
problem Improving RL exploration without compromising stability and expressiveness.
method Disentangled actor-critic approach with analogous pairs of actors and critics.
result Empirical evaluation shows ADAC outperforms alternatives in challenging exploration tasks.
Reinforcement learning is a powerful paradigm for learning optimal policies from experimental data. However, to find optimal policies, most reinforcement learning algorithms explore all possible actions, which may be harmful for real-world systems. As a consequence, learning algorithms are rarely applied on safety-crit…
New TD method stabilizes average-reward learning.
problem Stability issues in average-reward TD learning.
method Implicit fixed point update for average-reward TD(λ). result Improved numerical stability and broader step-size range.
Cramming method evaluates learned policies from contextual bandits efficiently.
problem Evaluating final learned policies from contextual bandit algorithms.
method On-policy evaluation using a single pass of data, ensuring consistency and asymptotic normality.
result Cramming method reduces evaluation standard error by approximately 40% compared to off-policy methods.
Entropy regularization improves MFG learning efficiency and stability.
problem Improving Mean Field Game learning efficiency and stability.
method Entropy regularization applied to MFG with learning.
result Entropy regularization yields time-dependent policies and stabilizes convergence.
SFPO optimizes LLM reasoning by repositioning before updating, improving stability and efficiency.
problem Noisy gradients from low-quality rollouts cause instability and inefficient exploration in on-policy RL algorithms.
method Decomposes each step into three stages: a short fast trajectory, repositioning, and slow correction, preserving the objective and rollout process unchanged.
result SFPO consistently improves stability, reduces rollouts, and accelerates convergence, outperforming GRPO on math reasoning benchmarks.
The OGY method is one of control methods for a chaotic system. In the method, we have to calculate a stabilizing periodic orbit embedded in its chaotic attractor. Thus, we cannot use this method in the case where a precise mathematical model of the chaotic system cannot be identified. In this case, the delayed feedback…
Efficient algorithm reduces control system regret to sqrt(t).
problem Control of linear quadratic systems with guaranteed anytime performance.
method Combines SDP-based framework with regularization and perturbation mechanisms.
result Achieves anytime regret of order sqrt(t) with explicit dependence on system dimensions.
Dealing with high variance is a significant challenge in model-free reinforcement learning (RL). Existing methods are unreliable, exhibiting high variance in performance from run to run using different initializations/seeds. Focusing on problems arising in continuous control, we propose a functional regularization appr…
Emerging economies use countercyclical policies to manage crises and dominant currencies.
problem Managing economic crises and fluctuations in dominant currencies like USD and EUR.
method Theoretical analysis, case studies, econometric modeling.
result Emerging economies can stabilize growth with countercyclical monetary policies.
Study on sample complexity of policy gradient for stabilizing linear systems under multiplicative noise.
problem Learning optimal feedback gain for stabilizing linear systems with multiplicative noise.
method Analyzes the sample complexity of policy gradient methods, addressing the cusp obstruction and using symmetry to control divergent parts of the gradient.
result Proves that projected mini-batch policy gradient attains total sample complexity of O(1/η) when noise density is known and O(η^(-(2s+1)/(2s))) when estimated, for C^s noise densities with s ≥ 2.
New TD algorithms stabilize RL tasks by reformulating updates into fixed point equations.
problem TD learning's sensitivity to step size specification.
method Implicit TD algorithms reformulate TD updates into fixed point equations.
result Implicit TD algorithms are more stable and less sensitive to step size.
New method improves stability of soft FQI for offline RL.
problem Stability issues in soft FQI under function approximation.
method Stationary reweighting to align operator norms.
result Local linear convergence proved under certain conditions.
Overparameterized models generalize well in offline contextual bandits, but policy-based algorithms struggle.
problem The performance gap between value-based and policy-based algorithms in offline contextual bandits with overparameterized models.
method Analysis of action-stability in objectives and formal proofs of regret bounds.
result The performance gap is due to action-stability of objectives, with value-based objectives being stable and policy-based objectives unstable.
Risk-controlled post-processing optimizes decision policies under risk constraints.
problem Optimizing decision policies with risk constraints for better outcomes.
method Developed a post-processing algorithm that selects a threshold based on fitted fallback policy and score, leveraging tools from algorithmic stability and stochastic processes.
result The post-processed policy achieves precise expected risk control under exchangeability and meets or nearly meets risk budgets while preserving more agreement with the baseline.
A new RL framework handles autocorrelated actions for better learning and stability.
problem Improving reinforcement learning algorithms for better stability and efficiency.
method Introduces a new algorithm that optimizes policies with autocorrelated actions.
result The new algorithm outperforms existing methods in four simulated control problems.
SNPL learns safe policies for multi-objective interventions with high confidence.
problem Designing effective digital interventions balancing multiple objectives with noisy data.
method Leverages algorithmic stability to learn policies with high-confidence guarantees.
result Offers dramatic improvements in safety and policy gains with smaller sample sizes.
Paper improves TD learning algorithm bounds with linear approx.
problem Sharp bounds for TD method performance in MDPs.
method Polyak-Ruppert averaging, universal step size, refined error bounds, stability of random matrices.
result Near-optimal variance and bias terms achieved.
Paper proposes a new method to optimize robot body structure and control policy.
problem Optimizing robot body structure and control policy in a coupled manner.
method Revisits co-design problem as a Stackelberg game, incorporating control adaptation dynamics.
result Stackelberg PPO outperforms standard PPO in stability and performance.
Continuous control imitation learning fails if expert actions are smooth.
problem Continuous control imitation learning fails if expert actions are smooth.
method Study of imitation learning in discrete-time, continuous state-and-action control systems.
result Any smooth, deterministic imitator policy suffers exponentially larger error than the expert.
Adapts GRPO for off-policy RL, improving reward.
problem Improving training stability and efficiency in RL.
method Adapts GRPO to off-policy setting, uses clipped surrogate objectives.
result Off-policy GRPO outperforms on-policy GRPO in empirical tests.
In sequence generation task, many works use policy gradient for model optimization to tackle the intractable backpropagation issue when maximizing the non-differentiable evaluation metrics or fooling the discriminator in adversarial learning. In this paper, we replace policy gradient with proximal policy optimization (…
The paper improves reinforcement learning stability and efficiency with a new theoretical framework.
problem Stability and efficiency in reinforcement learning, especially in data-scarce scenarios.
method Theoretical framework using resampled U- and V-statistics to model experience replay, applied to policy evaluation and kernel ridge regression. result Significant improvements in stability and efficiency, particularly in data-scarce scenarios.
Hybrid controller combines model-based and policy-based reinforcement learning.
problem Combining model-based and policy-based reinforcement learning for stability and robustness.
method Designs a hybrid controller that interpolates a model-based linear controller and a differentiable policy.
result Proven to maintain stability and universal approximation properties.
Study examines Fed's pandemic communication strategies.
problem Analyzing Federal Reserve's communication during the COVID-19 pandemic.
method Sentiment analysis, topic modeling, comparative analysis of previous crises.
result Fed's communication during the pandemic focused on financial stability, market volatility, social welfare, and unconventional monetary policy.