Search-R1

Search-R1 is an open-source reinforcement learning (RL) framework, released under the Apache 2.0 license, that trains large language models to autonomously generate search queries and interleave retrieval with step-by-step reasoning. It was developed by researchers at the University of Illinois Urbana-Champaign, UMass Amherst, and Google Cloud AI Research, and is built on top of the veRL training library. The codebase and model checkpoints are publicly available on GitHub.

What It Is

Search-R1 extends the DeepSeek-R1(-Zero) paradigm by adding interleaved search engine access to the RL training loop. Rather than prompting an LLM to use a search engine at inference time—which the paper notes is often suboptimal—Search-R1 trains the model end-to-end with RL so it learns when and how to call a search engine as part of its reasoning trajectory. The framework supports PPO, GRPO, and REINFORCE as RL algorithms, and is compatible with multiple LLM families (Llama 3, Qwen 2.5, and others) and multiple retrieval backends (local sparse BM25, local dense retrievers with flat or ANN indexing, and online APIs such as Google, Bing, and Brave Search).

How the Training Loop Works

During RL rollout, the LLM generates tokens and can emit special <search> and </search> tokens to trigger a retrieval call. The retrieved passages are injected into the sequence wrapped in <information> and </information> tokens, and the model continues reasoning inside <think> and </think> blocks before producing a final answer in <answer> and </answer> tags. A key design choice is retrieved token loss masking: the policy gradient objective is computed only over LLM-generated tokens, not over retrieved content, which the paper reports stabilizes training and improves final performance. The reward function is a simple rule-based exact-match outcome reward, avoiding the complexity of neural reward models.

Empirical Results

The paper reports that Search-R1 achieves average relative improvements of 24% (Qwen2.5-7B) and 20% (Qwen2.5-3B) over RAG baselines across seven question-answering datasets, covering both general QA (NQ, TriviaQA, PopQA) and multi-hop QA (HotpotQA, 2WikiMultiHopQA, Musique, Bamboogle). The paper also finds that PPO provides more stable training than GRPO (which can exhibit reward collapse over long runs), while GRPO converges faster. Larger models show greater gains from search integration than smaller ones.

Ecosystem and Adoption

The GitHub repository lists a growing set of downstream projects that build on or are inspired by Search-R1, including DeepResearcher, Multimodal-Search-R1, ZeroSearch, SkyRL, and verl-tool, among others. As of the available data, the repository has over 5,200 stars and 466 forks. Search-R1 has also been integrated into the veRL library's official multi-turn tool-use examples, and is featured in Thinking Machines Lab's Tinker product.

Update: Second Paper and Ongoing Development

A second paper (arXiv 2505.15117) conducting detailed empirical studies was published in May 2025, accompanied by new experiment logs (v0.3). Earlier milestones include the initial open-source release in February 2025 (preliminary results), the first paper in March 2025 (v0.1/v0.2 logs), and multinode training support for 30B+ LLMs added in April 2025. The project continues to receive updates, with the last push to the repository recorded in November 2025.