A new algorithm combines advantage actor-critic with value distribution for improved performance.
problem Improving reinforcement learning performance through better value function estimation.
method Developed a new algorithm (DA2C or QR-A2C) that uses value distribution estimated by quantile regression.
result Achieved at least as good as baseline algorithms and outperformed in some tasks with smaller variance and increased stability.
Improved A2C method with lower variance.
problem Reducing variance in deep policy gradient methods.
method Using control variate theory, derived a new A2C formulation with lower variance.
result New A2C method has lower variance and improved performance.
New algorithm learns deterministic policies in continuous domains using sign of advantage function.
problem Learning deterministic policies in continuous domains.
method Proposes a new trust region algorithm, Penalized NFAC (PeNFAC), based on a theoretical explanation of the advantage function.
result PeNFAC algorithm surpasses state-of-the-art algorithms in learning deterministic policies in classic control problems.
SIL learns to repeat past good decisions for better exploration.
problem Improving exploration in reinforcement learning environments.
method Self-Imitation Learning (SIL) is a simple off-policy actor-critic algorithm.
result SIL significantly improves A2C on Atari games and is competitive with state-of-the-art methods.
The paper tackles scalarization issues in A2C RL algorithms, proposing methods to avoid gradient overlap and noise.
problem Scalarization issues in A2C RL algorithms leading to gradient overlap and uncontrolled noise.
method Proposes techniques to avoid gradient overlap and noise in A2C RL algorithms.
result Pilot experiments show the proposed method speeds up training in A2C RL algorithms.
Improved SAC with AWMP for better control tasks.
problem Discontinuous and non-smooth optimal policies in reinforcement learning.
method Advantage Weighted Mixture Policy (AWMP) for SAC, learning state-specific weights.
result SAC with AWMP outperforms SAC in four control tasks.
Inspiration Learning expands imitation learning to different action spaces.
problem Current imitation learning techniques are limited to agents with the same action space.
method Designs Advantage Actor-Critic algorithms using Preferential based Reinforcement Learning.
result Method extends imitation learning to new action spaces.
TT-DAC-PS: A deterministic actor-critic approach for optimal trade execution
problem Optimal execution of large stock sell programs
method Twin-Target Deterministic Actor-Critic with Policy Smoothing
result Reduces mean implementation shortfall percentage
Paper proposes a more robust deep reinforcement learning agent.
problem Creating more robust reinforcement learning agents.
method Modified A3C algorithm with dual input streams.
result Significant reduction in training parameters (30%) with improved robustness.
Smaller actor-critic models lead to performance degradation and overfitting, highlighting the critic's role in value underestimation.
problem Performance degradation and overfitting in actor-critic models with smaller actors.
method Broad empirical investigations and analyses of asymmetric actor-critic setups, exploring techniques to mitigate value underestimation.
result Value underestimation is a key cause of performance degradation in smaller actor-critic models, and the critic plays a crucial role in mitigating this.
New method combines value function decomposition and policy gradients for cooperative multi-agent reinforcement learning.
problem Challenges in cooperative multi-agent reinforcement learning, especially credit assignment and large action spaces.
method Decomposed Soft Actor-Critic (mSAC) method with Q network architecture, discrete probabilistic policy, and counterfactual advantage function.
result Significantly outperforms policy-based approach COMA and achieves competitive results with SOTA value-based approach Qmix.
PEARL uses reinforcement learning to improve matrix preconditioners.
problem Learning effective preconditioners for iterative solvers is challenging.
method PEARL employs an actor-critic reinforcement learning framework to learn preconditioners dynamically.
result PEARL outperforms traditional and neural preconditioners in flexibility and solving speed.
Policy gradient is an efficient technique for improving a policy in a reinforcement learning setting. However, vanilla online variants are on-policy only and not able to take advantage of off-policy data. In this paper we describe a new technique that combines policy gradient with off-policy Q-learning, drawing experie…
FRD protects privacy in distributed RL by sharing proxy experience memory.
problem Privacy violation in exchanging experience memory in distributed RL.
method Proposes FRD framework using proxy experience memory.
result Numerical evaluation shows FRD is effective and performance depends on proxy memory structure.
Paper proposes Terminal Prediction to improve deep RL performance.
problem Sample inefficiency and convergence to locally optimal policies in deep reinforcement learning.
method Introduces a self-supervised auxiliary task, Terminal Prediction, to help representation learning.
result A3C-TP outperforms standard A3C in most domains and provides significant improvement in Pommerman.
WRAAC uses Wasserstein distance for robust reinforcement learning.
problem Lack of quantified robustness to system dynamics in existing reinforcement learning algorithms.
method Leverages Wasserstein distance to connect state disturbance to transition kernel disturbance, reducing infinite-dimensional optimization to a finite-dimensional problem.
result Designs a novel algorithm, WRAAC, that achieves robust reinforcement learning.
Deep RL agent reconfigures dynamic visual sensor networks efficiently.
problem Efficiently reconfigure dynamic visual sensor networks.
method Modified asynchronous advantage actor-critic framework with Relational Network module, trained in an abstract simulation environment.
result System validated using real-world inputs and preexisting algorithms.
Paper provides convergence guarantees for off-policy NAC with finite sample complexity.
problem Convergence analysis of off-policy natural actor-critic algorithm.
method Finite-sample analysis with Importance Sampling and Q-trace algorithm.
result Converges to global optimal policy with sample complexity O(ε−3log2(1/ε)). Capsule Networks improve autonomous navigation in sparse environments.
problem Challenges in reinforcement learning for sparse reward environments.
method Caps-EM pairs CapsNets with Advantage Actor Critic, using fewer parameters.
result Caps-EM achieves significant time improvements and fewer parameters compared to competing methods.
Proposes tail-adaptive f-divergences for better inference.
problem Inference difficulties with heavy-tailed importance weights.
method Tail-adaptive f-divergences that change convex function with importance weights tail.
result Significant advantages over classical KL and α-divergences.
AR-A3C improves A3C's robustness to noisy environments.
problem Neural networks are vulnerable to noise from unexpected sources in reinforcement learning.
method Introduced an adversarial agent to make the learning process more robust.
result AR-A3C outperforms A3C in both clean and noisy environments.
IDS improves reinforcement learning with contextual information.
problem Optimizing IDS for contextual reinforcement learning.
method Investigated contextual bandit problems and proposed a computationally-efficient IDS.
result Contextual IDS outperforms conditional IDS by considering future contexts.
MODULE solves LfO problem with high sample efficiency and stability.
problem Sample inefficiency and instability in LfO.
method Modified DSAC with MODULE algorithm.
result Superior performance in MuJoCo environments.
The study uses AI to optimize trading in FX markets by considering size-dependent fees and risk-aversion.
problem Optimizing trading in FX markets with size-dependent fees and risk-aversion.
method Fitted Natural Actor-Critic (FNC) Reinforcement Learning algorithm.
result The algorithm effectively trades with variable order sizes, reducing transaction costs and promoting risk-averse behavior.
Study uses reinforcement learning to optimize portfolios under recursive utility.
problem Improving portfolio allocation using risk-sensitive objectives.
method Approximated certainty equivalent via Monte Carlo, trained actor-critic algorithms (PPO, A2C).
result Recursive-utility agent outperforms discounted baseline in Sharpe ratio, max drawdown, and cumulative return.
Paper tackles offline meta-reinforcement learning with a new algorithm.
problem Performing reinforcement learning on limited data from a new task.
method Meta-Actor Critic with Advantage Weighting (MACAW) algorithm.
result Achieves notable gains over prior methods on offline meta-RL benchmarks.
This work enhances curiosity-driven exploration in reinforcement learning.
problem Sparse or insufficient extrinsic feedback in real-world scenarios.
method Probabilistic attention mechanism combined with Actor-Critic framework and ICM.
result New methods improve agent performance in reinforcement learning tasks.
Actor-critic converges globally in LQR with ergodic cost.
problem Theoretical understanding of actor-critic algorithm's global convergence.
method Nonasymptotic convergence analysis of actor-critic in linear quadratic regulator (LQR) setting.
result Actor-critic finds globally optimal policy and value function at a linear rate.
New algorithm reduces bias in off-policy reinforcement learning.
problem Challenges in designing off-policy reinforcement learning algorithms.
method Doubly robust off-policy actor-critic (DR-Off-PAC) with a single timescale structure.
result Establishes the first overall sample complexity analysis for a single time-scale off-policy AC algorithm.
Deep RL multi-task learning outperforms single-task learning on new tasks.
problem Improving performance on new tasks in multi-task reinforcement learning.
method Investigation of multi-task reinforcement learning algorithms with and without Elastic Weight Consolidation (EWC).
result Multi-task reinforcement learning algorithms outperform single-task learning on new tasks.
A human-like strategy improves DRL performance in Atari games.
problem Lack of exploration in DRL for high-dimensional problems.
method Mixed strategy approach mimicking human behavior.
result Higher maximum score probability achieved in Breakout game.
The paper analyzes an actor-critic algorithm with target networks for deep reinforcement learning.
problem Lack of theoretical understanding of target networks in actor-critic methods.
method Proposes a theoretical analysis of an online target-based actor-critic algorithm with linear function approximation.
result Establishes asymptotic convergence results and finite-time analysis for both critic and actor.
A2C-based MARL controls large-scale traffic signals more efficiently.
problem Scalability issue in centralized RL for large-scale traffic control.
method Decentralized Multi-Agent A2C with improved observability and reduced learning difficulty.
result Optimal, robust, and sample-efficient control over other algorithms.
This work analyzes actor-critic methods for faster convergence.
problem Finite-time analysis and sample complexity of two-time-scale actor-critic methods.
method Non-asymptotic analysis under non-i.i.d. setting, proving convergence to first-order stationary point.
result Actor-critic method finds a first-order stationary point with ildeO(ε−2.5) sample complexity. New Soft Actor-Critic for discrete actions.
problem Applying reinforcement learning to games with discrete actions.
method Derived an alternative Soft Actor-Critic for discrete actions.
result Competitive with tuned model-free state-of-the-art on Atari games.
Improves stability of actor-critic methods by penalizing TD error.
problem Instability in actor-critic methods during learning.
method Regularize actor's learning objective by penalizing critic's TD error.
result Improves stability and overall performance of actor-critic methods.
New PAC-Bayesian approach stabilizes actor-critic learning.
problem Training instability in actor-critic algorithms.
method Employing PAC-Bayesian bound as the critic training objective.
result Significant improvement in online learning performance.
Paper proves convergence of actor-critic methods for MDPs.
problem Finding approximate solutions to entropy-regularized MDPs.
method Stochastic approximation and policy evaluation with general regularizers.
result Probability one convergence of actor-critic algorithms.
New analysis shows actor-critic method converges efficiently in practical settings.
problem Understanding finite-time convergence of single-timescale actor-critic methods.
method Investigated online single-timescale actor-critic algorithm with linear function approximation and Markovian sampling.
result Proved convergence to ε-approximate stationary point with sample complexity of O(ε^(-2)).
KGRL uses reinforcement learning with knowledge graphs for better interactive recommendation.
problem Achieving responsiveness and accuracy in dynamic user-item interactions.
method KGRL combines reinforcement learning and knowledge graphs, using a local knowledge network and attention mechanism.
result KGRL outperforms state-of-the-art methods in simulated and real-world environments.
Hybrid actor-critic learns in complex action spaces.
problem Learning in complex, structured action spaces.
method Parallel sub-actor networks and a critic network.
result Hybrid PPO outperforms previous methods in parameterized action spaces.
DAC combines actor-critic methods for learning options.
problem Learning hierarchical policies in reinforcement learning.
method Double Actor-Critic (DAC) architecture using two parallel augmented MDPs.
result DAC outperforms existing methods in robot simulation tasks.
This paper pretrains actor-critic RL algorithms using expert demonstrations.
problem Stability and efficiency of pretraining methods for actor-critic RL algorithms.
method Employ expert demonstrations in actor-critic reinforcement learning framework, ensuring non-global optimal demonstrations are used.
result Our method outperforms RL algorithms without pretraining and is more simulation efficient.
We present an actor-critic framework for MDPs where the objective is the variance-adjusted expected return. Our critic uses linear function approximation, and we extend the concept of compatible features to the variance-adjusted setting. We present an episodic actor-critic algorithm and show that it converges almost su…
WAVE improves stability in reinforcement learning by adaptively weighting critic's loss.
problem Inherent instability in actor-critic reinforcement learning algorithms.
method Wasserstein adaptive value estimation with Sinkhorn approximation.
result Achieves $\mathcal{O}\left(\frac{1}{k}
ight)$ convergence rate for critic's mean squared error.
Single-timescale actor-critic finds globally optimal policy.
problem Finding globally optimal policy in reinforcement learning.
method Simultaneous actor and critic updates with linear or deep neural network approximations.
result Actor sequence converges to globally optimal policy at O(K−1/2) rate. Deep reinforcement learning boosts commodities trading performance.
problem Improving algorithmic trading performance in commodities markets.
method Formulated as a stochastic dynamical system, employed actor-based and actor-critic-based policy gradient algorithms with CNN and LSTM function approximators.
result DRL models increase Sharpe ratio by 83% compared to buy-and-hold.
Efficient actor-critic learning with shared experience replay improves data efficiency.
problem Challenges in actor-critic reinforcement learning with experience replay and off-policy learning stability.
method Combining actor-critic algorithms with shared experience replay, analyzing V-trace, proposing a trust region scheme.
result State-of-the-art data efficiency on Atari achieved with 200M environment frames.