Proximal Policy Optimization Suffices for On-Policy Reinforcement Learning

摘要

Proximal Policy Optimization (PPO) has been applied to a wide range of applications, from robotics to large language models (LLMs). Yet, despite its success, it still suffers from unstable updates, performance collapse, and extreme sensitivity to hyperparameters. Many alternatives have been proposed to alleviate these issues, often demonstrating empirically stronger performance on standard reinforcement learning (RL) benchmarks. In this work, we observe that PPO’s training instability fundamentally stems from the brittle inter-coupling between its surrogate optimization hyperparameters which jointly dictate the effective size of policy updates. This complex coupling makes tuning quite difficult and, in practice, often leaves PPO under-performing. To mitigate this, we introduce a criterion on trajectories generated by behavior and target policies which can be used as an early stopping rule. This formulation effectively decouples the optimization dynamics, allowing the algorithm to adaptively scale its updates using a single threshold parameter. We additionally find a scaling law between this threshold parameter and the number of trajectory batches collected, further reducing the tuning burden. Using this simple criterion, we evaluate PPO on continuous control tasks from the DM Control Suite, outperforming prior policy gradient methods and revealing that many reported improvements were largely artifacts of under-performing PPO baselines.