# OpenRLHF

> Open-source RLHF and agentic reinforcement learning framework for LLMs and VLMs built on Ray, vLLM and DeepSpeed.

OpenRLHF is an open-source Python framework for reinforcement learning from human feedback and agentic RL on large language and vision-language models. It combines a Ray + vLLM distributed architecture with DeepSpeed training and an agent-based execution pipeline. The project is released under the Apache 2.0 license, and its documentation says it is under active development.

## What It Is

OpenRLHF is a training framework for aligning and fine-tuning language models. It covers supervised fine-tuning (SFT), reward model training, direct preference optimization (DPO/IPO/cDPO), and RL training with PPO, REINFORCE++, REINFORCE++-baseline, GRPO, Dr. GRPO, RLOO and FlashREINFORCE. Models load directly from HuggingFace checkpoints, and training is driven by command-line scripts or Ray job submissions. The README and docs describe it as the first framework of its kind, which is a project claim.

## How the Architecture Works

Ray schedules the Actor, Critic, Reward and Reference models across GPUs, and vLLM handles sample generation. The README states that RLHF training spends about 80% of its time on sample generation. DeepSpeed ZeRO-3 handles memory-efficient training. A Hybrid Engine mode can colocate all models and vLLM engines on the same GPUs using sleep-mode memory sharing, which suits smaller clusters. Multi-node runs are supported through Ray and SLURM.

## Agent-Based Execution

Every RL run goes through a token-in-token-out agent pipeline, which separates the execution mode from the RL algorithm. Single-turn mode supports standard RLHF, remote HTTP reward models and custom Python reward functions for reinforced fine-tuning with rule-based or verifiable rewards. Multi-turn mode lets users implement reset/step environments, and an OpenAI-compatible server executor wraps vLLM to collect token traces. VLM RLHF, including multi-turn image feedback, was added in version 0.10.

## Throughput and Production Options

Async training and partial rollout overlap generation with training, with the docs noting a tradeoff in stability. Off-policy correction (TIS, ICEPOP, Seq-Mask-TIS) handles vLLM-to-training log-prob mismatches. Other options include DAPO dynamic filtering, sample packing, RingAttention for long context, LoRA/QLoRA, MoE support, and the Muon optimizer. Resumable checkpoints, best-checkpoint tracking, and Wandb and TensorBoard logging are also available.

## Setup Path

The README recommends an NVIDIA PyTorch Docker container, then `pip install openrlhf` with optional extras such as vllm, ring and liger. Installing from source is also supported. The documentation includes a quick start, CLI option reference, performance tuning, checkpointing and troubleshooting guides.

## Features
- Ray + vLLM distributed architecture
- PPO, REINFORCE++, REINFORCE++-baseline, GRPO, Dr. GRPO, RLOO and FlashREINFORCE
- Unified single-turn and multi-turn agent execution
- Hybrid Engine GPU colocation with sleep mode
- Async training and partial rollout
- Custom Python and remote HTTP reward functions
- Vision-Language Model RLHF
- SFT, reward model and DPO/IPO/cDPO training
- LoRA/QLoRA and MoE support
- Muon optimizer support
- Off-policy correction (TIS, ICEPOP)
- DAPO dynamic filtering
- RingAttention sequence parallelism
- Checkpoint recovery and best-checkpoint tracking
- Wandb and TensorBoard logging
- SLURM multi-node training

## Integrations
Ray, vLLM, DeepSpeed, HuggingFace Transformers, Weights & Biases, TensorBoard, SLURM, NVIDIA Docker, FlashAttention

## Platforms
LINUX, API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
v0.11.2

## Links
- Website: https://openrlhf.readthedocs.io/
- Documentation: https://openrlhf.readthedocs.io/en/latest/
- Repository: https://github.com/OpenRLHF/OpenRLHF
- EveryDev.ai: https://www.everydev.ai/tools/openrlhf