NVIDIA NeMo-Aligner

NeMo-Aligner is an NVIDIA open-source toolkit for aligning language models using the NeMo Framework. It supports algorithms such as SteerLM, Direct Preference Optimization, and Reinforcement Learning from Human Feedback. The repository is archived: as of 5/15/2025 it is no longer actively maintained, and NVIDIA recommends NeMo RL instead.

What It Is

NeMo-Aligner is a Python toolkit for end-to-end model alignment, meant to make language models safer, more helpful, and less harmful. It is built on the NeMo Framework, which scales training across thousands of GPUs using tensor, data, and pipeline parallelism for all alignment components. Checkpoints are cross-compatible with the NeMo ecosystem, which supports inference deployment and further customization.

Supported Alignment Methods

The README lists the following methods:

• Supervised fine-tuning
• Reward model training
• SteerLM, an attribute-conditioned SFT approach presented as a user-steerable alternative to RLHF
• RLHF with PPO
• RLHF with REINFORCE
• Direct Preference Optimization
• Self-Play Fine-Tuning (SPIN)

The README says RLHF pipelines gained accelerated generation through TensorRT-LLM. It also cites several NVIDIA models aligned with the toolkit, including Llama3-70B variants and Llama-3.1-Nemotron-70B-Instruct. The Nemotron-4-340B Instruct and Reward models were trained with it.

Setup Path

NeMo-Aligner comes included in NeMo containers, which run on machines with NVIDIA GPUs and drivers. It can also be installed with pip after installing NeMo, and requirements match the NeMo Toolkit plus PyTriton. An official Dockerfile is provided, and releases ship with pre-built containers.

Current Status

The repository is archived, and the latest listed release is v0.7.0, published 2025-03-12. NVIDIA points users to NeMo RL, a post-training library with Hugging Face integration, Megatron Core optimizations, and Ray scheduling. The code is licensed under Apache 2.0.