PPO's gradients are heavy-tailed, affecting learning; a robust estimator improves performance.
problem Heavy-tailedness of PPO gradients causing learning issues.
method Characterized heavy-tailed gradients, identified likelihood ratios and advantages as sources, proposed GMOM as a robust estimator.
result GMOM improves PPO performance without clipping tricks.