This work explores representation complexity in RL paradigms, revealing model-based RL as the easiest task.
problem Investigating the representation complexity gap among model-based, policy-based, and value-based RL.
method Demonstrated through analysis of Markov decision processes (MDPs) and introduced new classes of MDPs.
result Representation complexity hierarchy: model-based RL > policy-based RL > value-based RL.
This paper surveys RL for combinatorial optimization, focusing on TSP.
problem Optimizing solutions for combinatorial optimization problems.
method Reinforcement learning applied to combinatorial optimization problems, specifically the TSP.
result Deep learning mechanisms enhance RL algorithms for near-optimal solutions.
Optimistic RL algorithms are simplified for deep RL with competitive performance.
problem Achieving accurate optimism in model-based RL for large-scale problems.
method Interpreting scalable optimistic model-based algorithms as solving a tractable noise augmented MDP.
result Competitive regret bound of i l d e O ( ∣ S ∣ H ∣ A ∣ T ) ilde{\mathcal{O}}( |\mathcal{S}|H\sqrt{|\mathcal{A}| T } ) i l d e O ( ∣ S ∣ H ∣ A ∣ T ) for Gaussian noise augmentation. Mixed RL improves RL efficiency with dual representations.
problem Poor sampling efficiency in RL methods.
method Uses dual representations of environmental dynamics to search optimal policies.
result Proves convergence and recursive stability of the mixed RL.
Automates RL with sample-efficient hyperparameter optimization.
problem Challenges in applying deep RL due to hyperparameter sensitivity and inefficiency.
method Population-based AutoRL framework for meta-optimizing RL algorithms and architectures.
result Reduces the number of environment interactions needed for meta-optimization by up to an order of magnitude.
Paper establishes lower bounds and optimal algorithms for deployment-efficient RL.
problem Deployment efficiency in reinforcement learning.
method Optimization with constraints, lower bounds, algorithms.
result Established optimal algorithms for deployment-efficient RL.
RH-UCRL combines pessimism and optimism for robust RL.
problem Ensuring reliable performance in real-world RL tasks with worst-case scenarios.
method RH-UCRL is a model-based RL algorithm that optimizes between an agent and an adversary, distinguishing between epistemic and aleatoric uncertainty.
result RH-UCRL achieves near-optimal sample complexity guarantees and outperforms other robust RL algorithms in adversarial environments.
M3PO improves model-based meta-RL with theoretical guarantees.
problem Improving sample efficiency in multi-task RL with theoretical guarantees.
method Extending Janner et al. (2019) theorems, proposing M3PO with performance guarantees.
result M3PO outperforms existing methods in continuous-control benchmarks.
MOReL learns offline RL policies using pessimistic MDPs.
problem Offline RL's data efficiency and velocity.
method Two-step process: learn P-MDP and near-optimal policy in it.
result MOReL is minimax optimal and matches state-of-the-art results.
This study explains RL training dynamics in LLMs, focusing on token-level optimization and reasoning pattern reshaping.
problem Understanding the training dynamics of RL in LLMs to improve their reasoning capabilities.
method Empirical analysis and theoretical modeling of RL training process, focusing on reasoning patterns and token optimization.
result RL primarily optimizes a sparse subset of critical tokens, reshaping reasoning pattern distributions and affecting model performance.
Myopic optimization outperforms reinforcement learning in portfolio management, leading to lower returns and higher risks.
problem Reinforcement learning strategies in portfolio management yield lower or negative returns and higher risks compared to myopic optimization.
method Modeling execution/liquidation frictions with mark-to-market accounting, using Malliavin calculus to derive policy gradients and risk shadow price, and quantifying phantom profit.
result Myopic optimization outperforms reinforcement learning in portfolio management, leading to better returns and lower risks.
New RL algorithm minimizes distributional learning error.
problem Improving distributional reinforcement learning for better error minimization.
method Proposes a new model-based algorithm with theoretical minimax optimality.
result Proves minimax optimality for approximating return distributions.
Optimistic NPG improves policy optimization in online RL with efficient sample complexity.
problem Limited theoretical understanding of policy optimization, especially in online RL.
method Combines natural policy gradient with optimistic policy evaluation.
result Achieves optimal dimension dependence sample complexity for learning near-optimal policies.
Offline RL policies should adapt to unknown aspects of the environment.
problem Uncertainty in offline RL datasets leads to suboptimal policies.
method Adaptive policies that consider all transitions seen so far, solving an implicit POMDP.
result Optimal adaptive policies improve offline RL performance.
Paper connects RL and non-equilibrium statistical mechanics for entropy-regularized RL.
problem Obtaining analytical solutions for entropy-regularized RL.
method Mapping RL to non-equilibrium statistical mechanics, applying large deviation theory.
result Derives exact analytical results for optimal policy and dynamics in MDPs.
Unified framework for CO problems using RL, providing optimal solutions and convergence guarantees.
problem Combinatorial optimization problems
method Unified framework of Markov decision processes (MDPs) and value-based reinforcement learning (RL) techniques
result RL techniques converge to approximate solutions with a guarantee on optimality gap
ContraBAR uses contrastive learning to learn Bayes-optimal policies in RL.
problem Learning optimal policies for unknown tasks sampled from a known distribution.
method Proposes ContraBAR, a meta RL algorithm using contrastive predictive coding (CPC) for belief inference.
result ContraBAR achieves comparable performance to state-of-the-art methods and is computationally efficient.
Unified hybrid RL algorithm improves online RL performance with offline data.
problem Improving reinforcement learning performance with limited online data.
method A unified hybrid RL algorithm combining offline and online data.
result Unified algorithm achieves state-of-the-art results in sub-optimality gap and online learning regret.
RL framework optimizes trading costs in noisy markets.
problem Optimal execution and placement in noisy markets.
method Dual-window Denoise PPO RL network, imitation learning, comprehensive market features, flexible action formulation.
result RL agents outperformed TWAP strategy in execution cost.
Experimentally, it has been observed that humans and animals often make decisions that do not maximize their expected utility, but rather choose outcomes randomly, with probability proportional to expected utility. Probability matching, as this strategy is called, is equivalent to maximum entropy reinforcement learning…
Critic-regularized regression improves offline RL performance.
problem Poor performance of off-policy algorithms in offline RL.
method Critic-regularized regression (CRR) for policy learning from fixed datasets.
result CRR outperforms state-of-the-art offline RL algorithms significantly.
Survey of RL methods for optimizing power grid topologies.
problem Optimizing power grid operation with adaptive control strategies.
method Reinforcement Learning (RL) for dynamic and uncertain environments.
result Comprehensive evaluation of RL-based methods for power grid topology optimization.
New offline RL algorithms tackle partial data coverage with optimal performance and practicality.
problem Partial data coverage in offline RL datasets.
method Augmented Lagrangian method applied to MIS formulation for optimal offline RL.
result Statistically optimal offline RL with practical performance, eliminating conservatism.
Develops ODRPO to improve RL algorithms with better performance and stability.
problem RL algorithms converge to sub-optimal solutions due to limited policy representation.
method Integrates DRO approach to solve trust region constrained optimization problem without parameterizing policies.
result Achieves globally optimal policy update and higher sample efficiency.
To overcome the curses of dimensionality and modeling of Dynamic Programming (DP) methods to solve Markov Decision Process (MDP) problems, Reinforcement Learning (RL) methods are adopted in practice. Contrary to traditional RL algorithms which do not consider the structural properties of the optimal policy, we propose …
New algorithm reduces offline RL data requirements significantly.
problem Optimizing policies using only historical data in reinforcement learning.
method Off-Policy Double Variance Reduction (OPDVR) algorithm.
result OPDVR achieves optimal sample complexity with O ( H 2 / d m ε 2 ) O(H^2/d_mε^2) O ( H 2 / d m ε 2 ) episodes. To overcome the curse of dimensionality and curse of modeling in Dynamic Programming (DP) methods for solving classical Markov Decision Process (MDP) problems, Reinforcement Learning (RL) algorithms are popular. In this paper, we consider an infinite-horizon average reward MDP problem and prove the optimality of the th…
A new offline RL framework unifies imitation learning and vanilla offline RL.
problem Learning from expert datasets without active data collection.
method A new offline RL framework that interpolates between imitation learning and vanilla offline RL, using a weak concentrability coefficient and a lower confidence bound algorithm.
result LCB algorithm achieves a faster rate of 1 / N 1/N 1/ N for nearly-expert datasets, and is adaptively optimal for the entire data composition range. New algorithm for offline RL with linear approx in MDPs and MGs, nearly optimal.
problem Offline RL with linear function approximation in MDPs and MGs.
method Pessimism-based algorithm with uncertainty decomposition via reference function.
result Nearly minimax optimal performance in offline RL for MDPs and MGs.
This paper bridges offline and online RL by studying policy finetuning with a reference policy.
problem Sample-efficient reinforcement learning in online and offline settings.
method Design of policy finetuning algorithms and analysis of sample complexity.
result Theoretical analysis shows that the optimal policy finetuning algorithm is either offline reduction or purely online RL.
HyperAgent improves RL exploration in large-scale problems.
problem Efficient exploration in large-scale reinforcement learning problems.
method Hypermodel framework for incremental posterior approximation without conjugacy.
result HyperAgent achieves logarithmic per-step computational complexity and sublinear regret.
EQO uses a simple bonus term for efficient exploration in tabular RL.
problem Achieving minimax optimal reinforcement learning with practical efficiency.
method Exploration via Quasi-Optimism, employing a bonus proportional to state-action visit count.
result Achieves the sharpest known regret bound for tabular RL.
RL improves combinatorial optimization by automating heuristic search.
problem Hard combinatorial optimization problems with suboptimal solutions.
method Training RL agents to solve these problems.
result RL can outperform traditional algorithms in solving complex problems.
Rewriting history improves RL algorithms for solving multiple tasks.
problem Improving sample efficiency in multi-task reinforcement learning.
method Introducing hindsight relabeling as inverse RL to generalize goal-relabeling techniques.
result Relabeling data using inverse RL accelerates learning in multi-task settings.
Although reinforcement learning (RL) can provide reliable solutions in many settings, practitioners are often wary of the discrepancies between the RL solution and their status quo procedures. Therefore, they may be reluctant to adapt to the novel way of executing tasks proposed by RL. On the other hand, many real-worl…
Reinforcement learning (RL) algorithms have been successfully applied to a range of challenging sequential decision making and control tasks. In this paper, we classify RL into direct and indirect RL according to how they seek the optimal policy of the Markov decision process problem. The former solves the optimal poli…
Mirror descent method improved RL algorithms.
problem Improving RL algorithms for better performance.
method Mirror descent method applied to RL, solving trust-region problems.
result MDPO outperforms or matches other RL algorithms in continuous control tasks.
Fine-tuning RL with offline data reduces online interactions.
problem Optimizing RL with limited online interactions and offline data.
method Developed algorithm extsc{FTPedel} for MDPs with linear structure.
result Optimally reduces the number of online interactions needed.
MOPO optimizes offline RL by penalizing dynamics uncertainty.
problem Learning policies from offline data with distributional shift.
method Modify model-based RL to avoid distributional shift.
result MOPO outperforms model-free and standard model-based RL.
This tutorial reviews RL-based methods for optimizing diffusion models to maximize specific metrics.
problem Optimizing diffusion models to generate samples that maximize specific metrics in practical applications.
method Various RL algorithms including PPO, differentiable optimization, reward-weighted MLE, value-weighted sampling, and path consistency learning.
result Exploration of strengths and limitations of RL-based fine-tuning algorithms and their benefits compared to non-RL-based approaches.
ReSkill reconciles RL skill creation with policy optimization.
problem RL policies lack reusable strategies across tasks.
method Integrates skill creation into RL loop with three mechanisms.
result Consistently outperforms existing methods, especially on unseen tasks.
FMDP-BF algorithm improves RL in factored MDPs with exponential regret reduction.
problem Efficient reinforcement learning in factored MDPs with constrained RL.
method FMDP-BF algorithm leveraging factorization structure of FMDPs.
result FMDP-BF's regret is exponentially smaller than optimal algorithms for non-factored MDPs.
Hybrid RL algorithms improve offline and online RL in linear MDPs.
problem Improving RL performance without single-policy concentrability.
method Developed computationally efficient algorithms for PAC and regret-minimizing RL in linear MDPs.
result Achieved sharper error or regret bounds for linear MDPs.
Proposes RFQI for robust RL using offline data.
problem Learning robust policies in the presence of model uncertainty.
method RFQI algorithm using offline data to learn optimal robust policy.
result RFQI learns near-optimal robust policy under standard assumptions.
Designing optimal controllers continues to be challenging as systems are becoming complex and are inherently nonlinear. The principal advantage of reinforcement learning (RL) is its ability to learn from the interaction with the environment and provide optimal control strategy. In this paper, RL is explored in the cont…
Paper tackles RL for power grid topology optimization.
problem Managing large action spaces in growing power networks.
method Hierarchical multi-agent reinforcement learning (MARL) framework.
result MARL framework outperforms single-agent RL methods.
New RL algorithm tackles online robust MDPs with uncertainty.
problem Developing robust reinforcement learning models for real-world environments.
method Proposes a robust optimistic policy optimization algorithm for online robust MDPs.
result Establishes the first regret bound for online robust MDPs.
Paper proposes an RL algorithm to ensure policy performance guarantees.
problem Lack of performance guarantees for RL policies compared to baselines.
method Online model-free algorithm that ensures conservative exploration.
result Regret bound of i l d e O ( T ) ilde{\mathcal{O}}(\sqrt{T}) i l d e O ( T ) for both discrete and continuous spaces.