Very recently proximal policy optimization (PPO) algorithms have been proposed as first-order optimization methods for effective reinforcement learning. While PPO is inspired by the same learning theory that justifies trust region policy optimization (TRPO), PPO substantially simplifies algorithm design and improves da…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “PPO algorithm”
PPOS improves PPO by smoothing the surrogate objective function.
problem Performance instability and optimization inefficiency in PPO.
method Use of a functional clipping method instead of a flat clipping method.
result PPOS conducts more accurate updates and outperforms other PPO variants.
Proximal Policy Optimization (PPO) is a highly popular model-free reinforcement learning (RL) approach. However, we observe that in a continuous action space, PPO can prematurely shrink the exploration variance, which leads to slow progress and may make the algorithm prone to getting stuck in local optima. Drawing insp…
In sequence generation task, many works use policy gradient for model optimization to tackle the intractable backpropagation issue when maximizing the non-differentiable evaluation metrics or fooling the discriminator in adversarial learning. In this paper, we replace policy gradient with proximal policy optimization (…
A new algorithm RTO integrates DPO and PPO for RLHF, improving performance.
problem Learning from sparse, sentence-level rewards in RLHF.
method Modeling RLHF as MDP, introducing RTO that learns token-wise reward function.
result RTO outperforms PPO on benchmarks by 7.5 points on AlpacaEval 2 and 4.1 points on Arena-Hard.
PPO algorithm converges to global optimality in multi-agent reinforcement learning.
problem Designing statistical guarantees for policy optimization methods in multi-agent reinforcement learning.
method Leveraging a multi-agent performance difference lemma, a localized action value function is used as a descent direction for each local policy, leading to a multi-agent PPO algorithm.
result The multi-agent PPO algorithm converges to the globally optimal policy at a sublinear rate under standard regularity conditions.
A new trading model combines GARCH and PPO for better stock trading profits.
problem Limited performance of reinforcement learning in stock market trading.
method Parallel-network continuous trading model using GARCH and PPO.
result The model achieves more profit compared to traditional reinforcement learning methods.
Optimistic PPO variant solves linear MDPs with improved regret bound.
problem Understanding theoretical limits of PPO in linear MDPs.
method Proposes an optimistic variant of PPO for episodic adversarial linear MDPs with full-information feedback.
result Establishes a regret bound.
Revisits PPO design choices, exposing failure modes and proposing alternatives.
problem Failure modes of standard PPO in new environments.
method Revisits standard PPO design choices, exposes failure modes, and proposes alternative approaches.
result Alternative design choices prevent failure modes in new environments.
Proximal policy optimization (PPO) is one of the most successful deep reinforcement-learning methods, achieving state-of-the-art performance across a wide range of challenging tasks. However, its optimization behavior is still far from being fully understood. In this paper, we show that PPO could neither strictly restr…
Study shows code-level optimizations significantly impact deep RL algorithms.
problem Understanding the impact of implementation details on deep RL algorithms.
method Case study on PPO and TRPO, investigating the effects of code-level optimizations.
result Code-level optimizations are crucial for performance in deep RL algorithms.
Recent successful deep reinforcement learning algorithms, such as Trust Region Policy Optimization (TRPO) or Proximal Policy Optimization (PPO), are fundamentally variations of conservative policy iteration (CPI). These algorithms iterate policy evaluation followed by a softened policy improvement step. As so, they are…
In this paper we propose a hybrid architecture of actor-critic algorithms for reinforcement learning in parameterized action space, which consists of multiple parallel sub-actor networks to decompose the structured action space into simpler action spaces along with a critic network to guide the training of all sub-acto…
We introduce two approaches for combining neural evolution strategy (NES) and proximal policy optimization (PPO): parameter transfer and parameter space noise. Parameter transfer is a PPO agent with parameters transferred from a NES agent. Parameter space noise is to directly add noise to the PPO agent`s parameters. We…
In importance sampling (IS)-based reinforcement learning algorithms such as Proximal Policy Optimization (PPO), IS weights are typically clipped to avoid large variance in learning. However, policy update from clipped statistics induces large bias in tasks with high action dimensions, and bias from clipping makes it di…
Proximal policy optimization(PPO) has been proposed as a first-order optimization method for reinforcement learning. We should notice that an exterior penalty method is used in it. Often, the minimizers of the exterior penalty functions approach feasibility only in the limits as the penalty parameter grows increasingly…
PPO's gradients are heavy-tailed, affecting learning; a robust estimator improves performance.
problem Heavy-tailedness of PPO gradients causing learning issues.
method Characterized heavy-tailed gradients, identified likelihood ratios and advantages as sources, proposed GMOM as a robust estimator.
result GMOM improves PPO performance without clipping tricks.
Deep RL algorithms struggle with noisy rewards in portfolio optimisation.
problem Evaluating deep reinforcement learning for portfolio optimisation with market impact.
method Simulated data with geometric Brownian motion and market impact model; Kelly criterion as upper bound; PPO and A2C with GAE; clipping; hidden Markov model for regime changes.
result PPO and A2C with GAE perform better with noisy rewards; PPO with HMM learns different policies for regime changes.
In this paper, a novel racing environment for OpenAI Gym is introduced. This environment operates with continuous action- and state-spaces and requires agents to learn to control the acceleration and steering of a car while navigating a randomly generated racetrack. Different versions of two actor-critic learning algor…
Model-free reinforcement learning methods such as the Proximal Policy Optimization algorithm (PPO) have successfully applied in complex decision-making problems such as Atari games. However, these methods suffer from high variances and high sample complexity. On the other hand, model-based reinforcement learning method…
Proximal policy optimization (PPO) is one of the most popular deep reinforcement learning (RL) methods, achieving state-of-the-art performance across a wide range of challenging tasks. However, as a model-free RL method, the success of PPO relies heavily on the effectiveness of its exploratory policy search. In this pa…
A new framework improves reinforcement learning algorithms with policy guarantees.
problem Designing efficient and stable reinforcement learning algorithms.
method A general framework (FMA-PG) based on functional mirror ascent that constructs surrogate functions enabling policy improvement guarantees.
result The proposed framework enables policy improvement guarantees that hold regardless of policy parameterization, and recovers important heuristics.
AlphaGrad optimizes memory usage in RL algorithms by normalizing gradients.
problem Memory overhead and hyperparameter complexity in adaptive optimizers.
method Tensor-wise L2 normalization followed by a smooth hyperbolic tangent transformation controlled by a single parameter.
result AlphaGrad provides enhanced training stability and competitive performance in various RL algorithms.
TOPPO improves PPO for MTRL by balancing critic gradients, outperforming SAC.
problem Critic-side gradient ill-conditioning in PPO for MTRL.
method Critic Balancing modules to improve gradient conditioning and balance task updates.
result TOPPO achieves stronger mean and tail-task performance than SAC-family and ARS-family baselines.
Hybrid LSTM-PPO optimizes dynamic portfolios with better performance.
problem Dynamic portfolio optimization under non-stationary market conditions.
method Combines LSTM for forecasting and PPO for adaptive portfolio adjustments.
result Hybrid framework outperforms single-model and equal-weight approaches in various metrics.
Paper compares RL models for finance, finding Reward Clipping best.
problem Optimizing asset allocation in finance.
method Actor-only, actor-critic, and PPO models compared; Reward Clipping introduced.
result Reward Clipping model outperforms others in bull and bear markets.
Adaptive Alpha Weighting with PPO: Enhancing Prompt-Based LLM-Generated Alphas in Quant Tradingcs.CE
PPO optimizes LLM-generated alpha weights for better trading performance.
problem Adapting LLM-generated alphas for varying market conditions.
method Proximal Policy Optimization (PPO) for dynamic alpha weight adjustment.
result PPO-optimized strategy achieves higher Sharpe ratios and smaller drawdowns.
In this paper, we implement three state-of-art continuous reinforcement learning algorithms, Deep Deterministic Policy Gradient (DDPG), Proximal Policy Optimization (PPO) and Policy Gradient (PG)in portfolio management. All of them are widely-used in game playing and robot control. What's more, PPO has appealing theore…
Develops ODRPO to improve RL algorithms with better performance and stability.
problem RL algorithms converge to sub-optimal solutions due to limited policy representation.
method Integrates DRO approach to solve trust region constrained optimization problem without parameterizing policies.
result Achieves globally optimal policy update and higher sample efficiency.
Study proposes adaptive RL for dynamic portfolio optimization.
problem Traditional portfolio optimization models fail to adapt to regime shifts.
method Regime-aware reinforcement learning framework with hybrid observations and constrained reward functions.
result Transformer PPO achieves highest risk-adjusted returns, while LSTM variants offer a good balance.
An Empirical Analysis of Proximal Policy Optimization with Kronecker-factored Natural Gradientscs.AI
In this technical report, we consider an approach that combines the PPO objective and K-FAC natural gradient optimization, for which we call PPOKFAC. We perform a range of empirical analysis on various aspects of the algorithm, such as sample complexity, training speed, and sensitivity to batch size and training epochs…
As the most successful variant and improvement for Trust Region Policy Optimization (TRPO), proximal policy optimization (PPO) has been widely applied across various domains with several advantages: efficient data utilization, easy implementation, and good parallelism. In this paper, a first-order gradient reinforcemen…
This paper analyzes DRL strategies in finance, revealing unique trading patterns and performance differences.
problem Limited research on DRL behavior in finance applications.
method Analysis of trading behaviors and purchase diversity of DRL algorithms (A2C, PPO, SAC, DDPG, TD3).
result DRL algorithms exhibit distinct trading patterns and performance differences, with A2C outperforming others in terms of cumulative rewards.
GPG improves RL from feedback with less memory and compute.
problem Improving reinforcement learning from human feedback.
method Group-based Monte Carlo advantage estimator replacing learned value function.
result GPG matches or outperforms PPO on standard benchmarks.
New algorithms learn graphons in GMFGs without knowing them.
problem Learning graphons in Graphon Mean-Field Games with unknown graphons.
method Proximal Policy Optimization for GMFG (GMFG-PPO) and kernel embedding methods for estimating graphons.
result The proposed algorithms reduce exploitability when learning unknown graphons.
Proximal policy optimization and trust region policy optimization (PPO and TRPO) with actor and critic parametrized by neural networks achieve significant empirical success in deep reinforcement learning. However, due to nonconvexity, the global convergence of PPO and TRPO remains less understood, which separates theor…
This paper compares eight DRL algorithms for dynamic hedging.
problem Optimal dynamic hedging strategies using Deep Reinforcement Learning.
method Eight DRL algorithms (MCPG, PPO, DQL, DDPG) compared using a GJR-GARCH(1,1) simulated dataset.
result MCPG and PPO outperform the Black-Scholes delta hedge baseline.
This paper proposes Self-Imitation Learning (SIL), a simple off-policy actor-critic algorithm that learns to reproduce the agent's past good decisions. This algorithm is designed to verify our hypothesis that exploiting past good experiences can indirectly drive deep exploration. Our empirical results show that SIL sig…
RADIAL-RL improves deep RL agents' robustness against adversarial attacks.
problem Vulnerability of deep reinforcement learning agents to small adversarial perturbations.
method RADIAL-RL, a principled framework for training robust reinforcement learning agents.
result RADIAL-RL-trained agents consistently outperform prior methods in robustness tests.
Enhances portfolio performance using deep reinforcement learning and future rewards.
problem Improving existing high-performing portfolio strategies through dynamic rebalancing.
method Proximal Policy Optimization (PPO) and Oracle agents for dynamic rebalancing; Regret-based Sharpe reward function; Transaction cost scheduler; Future-looking reward function; Circular block bootstrap training.
result Significantly enhanced portfolio performance compared to traditional strategies and baselines.
Mirror descent method improved RL algorithms.
problem Improving RL algorithms for better performance.
method Mirror descent method applied to RL, solving trust-region problems.
result MDPO outperforms or matches other RL algorithms in continuous control tasks.
TROLL improves RL for LLMs by replacing clipping with a trust region projection.
problem Clipping in RL for LLMs causes instability and suboptimal performance.
method TROLL uses a discrete differentiable trust region projection to replace clipping, balancing computational cost and effectiveness.
result TROLL consistently outperforms PPO-like clipping in training speed, stability, and final success rates.
Paper proposes a new method to optimize robot body structure and control policy.
problem Optimizing robot body structure and control policy in a coupled manner.
method Revisits co-design problem as a Stackelberg game, incorporating control adaptation dynamics.
result Stackelberg PPO outperforms standard PPO in stability and performance.
The Pommerman simulation was recently developed to mimic the classic Japanese game Bomberman, and focuses on competitive gameplay in a multi-agent setting. We focus on the 22 team version of Pommerman, developed for a competition at NeurIPS 2018. Our methodology involves training an agent initially through imit…
QuantFactor REINFORCE: Mining Steady Formulaic Alpha Factors with Variance-bounded REINFORCEq-fin.CP
Paper proposes a new REINFORCE algorithm for mining formulaic alpha factors with reduced variance.
problem Mining formulaic alpha factors with interpretability and robustness in volatile markets.
method Developed a novel REINFORCE algorithm with a dedicated baseline and reward shaping.
result Boosts correlation with returns by 3.83% and enhances excess returns compared to existing methods.
AI algorithms outperform traditional trading methods in stock markets.
problem Traditional trading methods struggle with risk management and edge over classical approaches.
method Used Deep Reinforcement Learning (DRL) algorithms (DDQN and PPO) to compare with Buy and Hold benchmark.
result DRL algorithms provide a substantial edge over classical approaches in terms of risk-adjusted returns.
Modulating masks improve lifelong reinforcement learning.
problem Catastrophic forgetting and task interference in lifelong reinforcement learning.
method Adapted modulating masks for deep lifelong reinforcement learning (LRL) with PPO and IMPALA agents.
result Superior performance in both discrete and continuous RL tasks compared to LRL baselines.
New method improves DRL robustness against adversarial state observations.
problem Adversarial attacks on deep reinforcement learning agents observing state data.
method State-adversarial Markov decision process (SA-MDP) and policy regularization.
result Significant improvement in robustness of DRL algorithms under adversarial attacks.