verl

verl is an open-source reinforcement learning training library for large language models, initiated by the ByteDance Seed team and maintained by the verl community. It is the open-source version of the HybridFlow RLHF framework paper and is released under the Apache License 2.0. The latest release listed on the repository is v0.9.1.

What It Is

verl is a post-training library for running supervised fine-tuning and reinforcement learning on LLMs and vision-language models. Its hybrid-controller programming model lets users express RL dataflows such as GRPO and PPO in a few lines of code. It decouples computation and data dependencies so it can plug into existing LLM infrastructure, and it supports flexible placement of models onto different sets of GPUs.

Training and Rollout Stack

FSDP, FSDP2 and Megatron-LM are supported for training, while vLLM, SGLang and HF Transformers handle rollout generation. The README lists algorithms including PPO, GRPO, GSPO, ReMax, REINFORCE++, RLOO, PRIME and DAPO, with model-based and function-based (verifiable) rewards. It also covers multi-turn rollouts with tool calling, LoRA RL, and multi-modal RL. According to the README, it scales to 671B models and hundreds of GPUs using expert parallelism, and a 3D-HybridEngine reduces memory redundancy when switching between training and generation. Experiment tracking works with wandb, swanlab, mlflow and tensorboard.

Hardware and Ecosystem

The README documents support for NVIDIA, AMD ROCm GPUs (MI300X, MI325X, MI355X) and Ascend hardware. The verl-project organization also publishes related projects built on verl, including verl-vla for vision-language-action post-training, VeRL-Omni for diffusion and omni-modal RL, verl-SpeCo for speculative decoding co-training, uni-agent for training long-horizon agents, and RL-Insight for observability. A project blog covers releases and case studies.

Getting Started

The documentation on Read the Docs provides installation and quickstart guides, a programming guide, PPO and GRPO algorithm pages, data and reward-function preparation steps, and a performance tuning guide. Algorithm recipes live in a separate verl-recipe repository.