AgentRL

AgentRL is an open-source framework from THUDM for scaling agentic reinforcement learning across multi-turn, multi-task settings. It combines a Ray-based training package with an environment deployment framework (controller and task workers) built on AgentBench, and includes scripts for evaluating API, local, and trained models. The repository is MIT licensed and accompanies a paper on arXiv.

What It Is

AgentRL is a Python toolkit for training LLM agents with reinforcement learning. It has two parts: a training framework in the trainer directory, and an environment deployment framework made of a controller and task workers. Users compose training routines from provided workers and components, and the paper's environments and training data are available through AgentBench FC.

How the training pipeline works

The example async trainer builds a fully asynchronous GRPO agentic training pipeline over a Ray cluster using three worker pools:

• Rollout workers explore tasks and stream trajectories.
• Actor workers compute gradients and update the policy.
• Reference workers keep a frozen baseline for KL regularisation.

Ray placement groups reserve CPU/GPU bundles, and a spawn helper wraps Ray actors in a collective handle that initialises NCCL process groups. A plugin mechanism streams updated weights from actor rank 0 to the rollout shards. A DistributedTaskManager turns task definitions into Ray jobs and groups multi-turn trajectories by group_id so they stay together.

Environment and evaluation stack

The controller, refactored in Go, routes requests to task workers, manages sessions, and offers a monitoring dashboard, with a stated goal of supporting up to 10,000 concurrent sessions. Task workers are FastAPI servers that manage environment lifecycles, build prompts, parse and execute agent responses, and score performance. Besides HTTP transport, a gRPC option lets the controller reach workers wherever they are deployed.

For evaluation, server_agent.py runs and resumes evaluations and supports cross sampling between models, and check.py summarises results. An experimental agentrl-eval package offers a more flexible interface for evaluating multiple models across multiple tasks.