Molt

Molt is an open-source RL framework from the NVIDIA-NeMo organization for training language and vision-language models with agentic reinforcement learning. It combines Ray for placement and async queues, vLLM for rollout, and NVIDIA AutoModel with FSDP2 for PyTorch-native training. The README describes about 9.2K lines of RL code that scale to 1T-class MoE models.

What It Is

Molt is a training framework for researchers who want to run SFT and online RL on models from a single, readable codebase. The agent is the program: reward is any Python written inside an Env or ChatAgent, such as graders, multi-turn tool environments, VLM environments or LLM-as-judge calls. The trainer is a single actor with an optional KL reference model. Entry points are molt.cli.train_sft for SFT and molt.cli.train_rl_ray for RL.

How the Agent Contract Works

Each RL run points at a Python module that exports an AgentRunner. With the Env path, the framework drives vLLM, tokenization and per-turn budgets, and the user's step() returns a Result with a reward. With the ChatAgent path, the user owns the loop through the OpenAI or Anthropic SDK against an auto-launched local server that captures token-exact traces. When context compaction rewrites a prompt prefix, the server starts a new token-exact segment, so one rollout can yield several segment trajectories sharing one reward. Reference agents are included for math and geo3k VLM tool use.

Algorithms and Scaling

Supported estimators are reinforce, reinforce_baseline, rloo, grpo, dr_grpo, gae (PPO) and on_policy_distill. Importance-sampling correction handles the train/rollout logprob mismatch under async rollout. For MoE models it offers router replay (R3) and router freezing. It also supports FlashREINFORCE, LoRA, and MTP speculative decoding for rollout. Parallelism options include tensor, expert and context parallelism plus optimizer CPU offload. The README cites DeepSeek-V3 at --fsdp.ep_size 256 as an example.

Setup Path

The recommended install is the provided Docker container, which bundles the CUDA 13 stack for A100, H100, H200, B200 and GB200 GPUs. A local install via pip install -e ".[vllm]" requires CUDA 13. Quick-start and Slurm recipes cover Qwen3, Qwen3.6 VLM, Nemotron-Omni distillation and GLM-5.2 runs. An optional automodel-slim backend branch is available.