OpenRLHF

OpenRLHF is an open-source Python framework for reinforcement learning from human feedback and agentic RL on large language and vision-language models. It combines a Ray + vLLM distributed architecture with DeepSpeed training and an agent-based execution pipeline. The project is released under the Apache 2.0 license, and its documentation says it is under active development.

What It Is

OpenRLHF is a training framework for aligning and fine-tuning language models. It covers supervised fine-tuning (SFT), reward model training, direct preference optimization (DPO/IPO/cDPO), and RL training with PPO, REINFORCE++, REINFORCE++-baseline, GRPO, Dr. GRPO, RLOO and FlashREINFORCE. Models load directly from HuggingFace checkpoints, and training is driven by command-line scripts or Ray job submissions. The README and docs describe it as the first framework of its kind, which is a project claim.

How the Architecture Works

Ray schedules the Actor, Critic, Reward and Reference models across GPUs, and vLLM handles sample generation. The README states that RLHF training spends about 80% of its time on sample generation. DeepSpeed ZeRO-3 handles memory-efficient training. A Hybrid Engine mode can colocate all models and vLLM engines on the same GPUs using sleep-mode memory sharing, which suits smaller clusters. Multi-node runs are supported through Ray and SLURM.

Agent-Based Execution

Every RL run goes through a token-in-token-out agent pipeline, which separates the execution mode from the RL algorithm. Single-turn mode supports standard RLHF, remote HTTP reward models and custom Python reward functions for reinforced fine-tuning with rule-based or verifiable rewards. Multi-turn mode lets users implement reset/step environments, and an OpenAI-compatible server executor wraps vLLM to collect token traces. VLM RLHF, including multi-turn image feedback, was added in version 0.10.

Throughput and Production Options

Async training and partial rollout overlap generation with training, with the docs noting a tradeoff in stability. Off-policy correction (TIS, ICEPOP, Seq-Mask-TIS) handles vLLM-to-training log-prob mismatches. Other options include DAPO dynamic filtering, sample packing, RingAttention for long context, LoRA/QLoRA, MoE support, and the Muon optimizer. Resumable checkpoints, best-checkpoint tracking, and Wandb and TensorBoard logging are also available.

Setup Path

The README recommends an NVIDIA PyTorch Docker container, then pip install openrlhf with optional extras such as vllm, ring and liger. Installing from source is also supported. The documentation includes a quick start, CLI option reference, performance tuning, checkpointing and troubleshooting guides.