Megatron-LM

Megatron-LM is NVIDIA's repository for training transformer models at scale. It contains two components: Megatron-LM, a reference example with pre-configured training scripts, and Megatron Core, a composable library of GPU-optimized building blocks. The latest documented docs version is 0.19.0, and the repository lists a core_v0.19.2 release.

What It Is

Megatron Core provides transformer building blocks, parallelism strategies (tensor, pipeline, data, expert, and context parallelism), mixed precision support (FP16, BF16, FP8, FP4), and model architectures. The README positions it for framework developers and ML engineers building custom training pipelines, while the Megatron-LM reference scripts target research teams, learning distributed training, and quick experimentation.

Training Workflow

After installing from PyPI (megatron-core) or from source, the quickstart first runs a minimal distributed loop with mock data on two GPUs to validate the environment. It then runs a LLaMA-3 8B training script with FP8 mixed precision on eight GPUs. To train on custom data, users prepare a JSONL file with a text field and run tools/preprocess_data.py to tokenize it into .bin and .idx binary files.

Advanced Capabilities

The documentation covers Mixture of Experts, multi-token prediction, multi-latent attention, context parallelism, Megatron-FSDP, a distributed optimizer, optimizer CPU offload, activation offloading, distributed checkpointing, inference engines, quantization, and reinforcement learning. Supported models include language and multimodal models. Megatron Bridge, a separate project, converts checkpoints between Hugging Face and Megatron formats.

Performance Claims

The README states the codebase trains models from 2B to 462B parameters across thousands of GPUs, reaching up to 47% Model FLOP Utilization on H100 clusters, measured without training to convergence.