# Molt

> A PyTorch-native, agentic-first reinforcement learning framework for research, built on Ray, vLLM and NVIDIA AutoModel.

Molt is an open-source RL framework from the NVIDIA-NeMo organization for training language and vision-language models with agentic reinforcement learning. It combines Ray for placement and async queues, vLLM for rollout, and NVIDIA AutoModel with FSDP2 for PyTorch-native training. The README describes about 9.2K lines of RL code that scale to 1T-class MoE models.

## What It Is

Molt is a training framework for researchers who want to run SFT and online RL on models from a single, readable codebase. The agent is the program: reward is any Python written inside an `Env` or `ChatAgent`, such as graders, multi-turn tool environments, VLM environments or LLM-as-judge calls. The trainer is a single actor with an optional KL reference model. Entry points are `molt.cli.train_sft` for SFT and `molt.cli.train_rl_ray` for RL.

## How the Agent Contract Works

Each RL run points at a Python module that exports an `AgentRunner`. With the `Env` path, the framework drives vLLM, tokenization and per-turn budgets, and the user's `step()` returns a `Result` with a reward. With the `ChatAgent` path, the user owns the loop through the OpenAI or Anthropic SDK against an auto-launched local server that captures token-exact traces. When context compaction rewrites a prompt prefix, the server starts a new token-exact segment, so one rollout can yield several segment trajectories sharing one reward. Reference agents are included for math and geo3k VLM tool use.

## Algorithms and Scaling

Supported estimators are reinforce, reinforce_baseline, rloo, grpo, dr_grpo, gae (PPO) and on_policy_distill. Importance-sampling correction handles the train/rollout logprob mismatch under async rollout. For MoE models it offers router replay (R3) and router freezing. It also supports FlashREINFORCE, LoRA, and MTP speculative decoding for rollout. Parallelism options include tensor, expert and context parallelism plus optimizer CPU offload. The README cites DeepSeek-V3 at `--fsdp.ep_size 256` as an example.

## Setup Path

The recommended install is the provided Docker container, which bundles the CUDA 13 stack for A100, H100, H200, B200 and GB200 GPUs. A local install via `pip install -e ".[vllm]"` requires CUDA 13. Quick-start and Slurm recipes cover Qwen3, Qwen3.6 VLM, Nemotron-Omni distillation and GLM-5.2 runs. An optional automodel-slim backend branch is available.

## Features
- Fully-async agentic RL with Ray, vLLM and NVIDIA AutoModel/FSDP2
- Gymnasium-style Env and ChatAgent agent APIs
- OpenAI- and Anthropic-compatible session server with token-exact trace capture
- Multi-turn, multimodal (VLM) and tool-use rollouts
- Estimators: reinforce, rloo, grpo, dr_grpo, gae (PPO), FlashREINFORCE
- On-policy distillation to a frozen teacher
- Importance-sampling correction for async rollout
- MoE router replay (R3) and router freeze
- TP / EP / CP parallelism and Adam CPU offload
- LoRA fine-tuning
- MTP speculative decoding for rollout
- SFT and RL CLI entry points with Slurm and quick-start recipes

## Integrations
Ray, vLLM, NVIDIA AutoModel, PyTorch, Hugging Face, OpenAI SDK, Anthropic SDK, Docker, Slurm, TransformerEngine, DeepEP

## Platforms
LINUX, WEB, API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
v0.1.11

## Links
- Website: https://github.com/NVIDIA-NeMo/labs-molt
- Repository: https://github.com/NVIDIA-NeMo/labs-molt
- EveryDev.ai: https://www.everydev.ai/tools/molt