SkyRL's FP8 Adoption Cuts Rollout Time by Up to 23%

SkyRL has announced that it's adopted FP8 for its reinforcement learning (RL) stack, leading to a significant reduction in rollout time by up to 23%. This breakthrough is particularly relevant for large language models and RLHF workloads.

The use of FP8 precision across training and rollout processes enables faster model convergence while reducing hardware overhead. According to SkyRL's blog post, FP8 cuts RL step times by up to 23% on NVIDIA Hopper (H100) and Blackwell (B200) GPUs, maintaining performance comparable to BF16.

The innovation hinges on SkyRL's on-policy weight synchronization (OPWS), which ensures the rollout engine operates with the exact FP8-trained policy. This addresses a challenge noted in other FP8 RL stacks, such as NVIDIA's NeMo RL and recent academic proposals like Jet-RL, which warned against instability caused by mismatched quantization between training and rollout phases.

FP8 offers significant advantages over BF16 and FP32 in RL workloads. It reduces memory traffic, lowers GPU memory requirements, and increases arithmetic throughput on tensor cores. SkyRL reports that FP8 parameter storage reduces per-GPU memory usage by 39-42%, freeing up resources for larger models or longer-context rollouts.