SkyRL Tames GPU Memory with FP8 Precision, Cuts Rollout Times by 23%

SkyRL has made a significant breakthrough in reinforcement learning (RL) efficiency by adopting FP8 precision across its training and rollout processes. According to Peter Zhang, SkyRL's innovation cuts RL step times by up to 23% on NVIDIA Hopper (H100) and Blackwell (B200) GPUs while maintaining performance on par with BF16.

The company's on-policy weight synchronization (OPWS) technology ensures that the rollout engine operates with the exact FP8-trained policy, avoiding numerical mismatch that previously plagued independent FP8 configurations. This refinement addresses a challenge noted in other FP8 RL stacks, such as NVIDIA's NeMo RL and recent academic proposals like Jet-RL.

FP8's role in RL efficiency lies in its ability to reduce memory traffic, lower GPU memory requirements, and increase arithmetic throughput on tensor cores. SkyRL reports that FP8 parameter storage reduces per-GPU memory usage by 39-42%, freeing up resources for larger models or longer-context rollouts.

The adoption of FP8 precision in SkyRL underscores a broader industry trend toward low-precision AI workloads, driven by NVIDIA's efforts to accelerate RL while preserving accuracy. The innovation has significant implications for the industry, particularly for large language models (LLMs) and RLHF (Reinforcement Learning with Human Feedback).

While challenges remain, including limitations on attention mechanisms and optimizer states still relying on higher precision, SkyRL's success with FP8 reinforces its potential to reshape reinforcement learning workflows.