rLLM

rLLM is an open-source framework for training language agents with reinforcement learning, developed as part of Berkeley Sky Computing Lab. It lets users bring an existing agent, score its outputs with a reward function, and update the underlying model, using the same agent code for evaluation and training. The latest listed release is v0.3.0-pre, published 2026-04-30.

What It Is

rLLM is a post-training framework for agentic programs. Users can wrap agent code built with LangGraph, the OpenAI Agents SDK, or plain openai.OpenAI using the @rllm.rollout decorator, define an evaluator as a reward function, and pass both to an AgentTrainer. A command-line interface covers model provider setup, benchmark evaluation (rllm eval), RL training (rllm train), and agent scaffolding (rllm init).

How the Training Pipeline Works

The documented pipeline is: run the agent, collect traces, compute rewards, update the model. A model gateway captures LLM calls, including token IDs and logprobs, and structures them into Episodes (one task) containing Trajectories (one agent run) made of Steps (one LLM call). A workflow engine runs parallel agent instances to collect rollouts, a transform pipeline groups trajectories for advantage computation, and a training backend performs the policy update.

Backends, Harnesses, and Benchmarks

Three training backends share one API and are switched with a flag: verl for distributed multi-GPU training, tinker for single-machine setups, and fireworks for the Fireworks platform. The README lists 10+ CLI harnesses (including Claude Code, Codex, and opencode) plus Harbor-compatible task directories. Rollouts can run in Docker, Daytona, Modal, or local sandboxes. It integrates 60+ benchmarks spanning math, code, QA, search, VLM, and agentic tasks, and supports methods such as GRPO, REINFORCE, RLOO, SFT, and on-policy distillation.

Setup Path

rLLM requires Python 3.11 or newer and installs via pip or uv, with optional extras for the verl and Fireworks backends; source and Docker installs are also documented. It is released under the Apache License 2.0. The project says models trained with it include rLLM-FinQA-4B, DeepSWE, DeepCoder-14B, and DeepScaleR-1.5B.