RAGEN

RAGEN (Reasoning AGENT) is an open-source reinforcement learning framework from the MLL Lab at Northwestern University for training reasoning LLM agents with multi-turn RL. It is built on StarPO (State-Thinking-Actions-Reward Policy Optimization) and ships with diagnostics for failure modes in agent RL training. RAGEN-2, released in March 2026, adds reasoning-collapse diagnostics and SNR-adaptive filtering.

What It Is

RAGEN is a training framework for researchers and practitioners who want to apply reinforcement learning to LLM agents that interact with environments over multiple turns. Agent-environment interactions are formulated as Markov decision processes in which states and actions are token sequences. The model produces reasoning-guided actions in a think/answer format, and the environment returns rewards and next states.

How Training Works

StarPO alternates between a rollout stage, where the LLM generates trajectories, and an update stage, where entire trajectories are optimized with importance sampling. It supports PPO with token-level advantage estimation and GRPO with trajectory-level normalized rewards. Ten built-in environments are included: Sokoban, FrozenLake, WebShop, DeepCoder, SearchQA, Lean, Bandit, Countdown, MetaMathQA, and Sudoku. A Gym-compatible interface makes it possible to add custom environments.

Diagnosing Reasoning Collapse

RAGEN-2 studies template collapse, where outputs look diverse within a single input but become input-agnostic across inputs, so entropy alone cannot detect it. It evaluates reasoning along two axes: conditional entropy, which measures within-input diversity, and mutual information, which measures cross-input distinguishability. Mutual-information proxy metrics are used to monitor collapse during training. SNR-adaptive filtering uses reward variance to keep high-signal prompts each iteration, and it can be configured as top-p or top-k rollout filtering. According to the project page, it improved results across RL algorithms, model scales, and modalities.

Setup Path

Installation uses a conda environment with Python 3.12 and a setup script, with an optional flag for the search environment. Training runs through a train.py entry point with config names such as the Sokoban config, and evaluation runs through the agent proxy module. The repository includes guides for evaluation, rollout filtering, MI metrics, and experiment reproduction. The code is released under the MIT license.