# NVIDIA NeMo Curator

> GPU-accelerated, open-source toolkit for building pipelines that load, filter, deduplicate, and transform text, image, video, and audio datasets for AI training.

NVIDIA NeMo Curator is an Apache 2.0-licensed Python toolkit from NVIDIA, part of the NVIDIA NeMo software suite. It helps ML engineers and data teams build repeatable, GPU-accelerated pipelines that prepare large text, image, video, and audio datasets for AI training. The same pipeline can run on a laptop or across a multi-node Ray cluster.

## What It Is

NeMo Curator is a data curation framework for LLM and multimodal training data. Pipelines are built from stages, each handling a discrete task such as load, filter, dedupe, classify, transform, or write. Modality plug-ins supply ready-made stages for text (deduplication, classification, quality filtering, language detection), image (aesthetic filtering, NSFW detection, embedding generation, deduplication), video (scene detection, clip extraction, motion filtering, deduplication), and audio (ASR transcription, quality assessment, WER filtering, speaker diarization).

## How Pipelines Execute

Stages declare their own resource requirements, such as CPU cores, GPU memory, and replicas. An executor runs the pipeline and auto-scales replicas per stage. The XennaExecutor (Cosmos-Xenna) is the production default, with experimental Ray-based backends also available. Streaming execution overlaps CPU and GPU work so stages run concurrently. It uses NVIDIA RAPIDS (cuDF, cuML, cuGraph) with Ray for multi-node, multi-GPU scaling.

## Recipes and Nemotron Connection

The README states that NeMo Curator powers the data pipelines behind NVIDIA Nemotron models, and that the Nemotron-CC recipe reproduces that dataset from Common Crawl, covering language ID, exact/fuzzy/substring deduplication, ensemble quality classification, and LLM-based synthetic data generation. The repository also includes tutorials for each modality, and an inference server feature that provides an OpenAI-compatible LLM endpoint inside a pipeline for synthetic data and classification workloads.

## Setup Path

Installation uses uv. A CPU path installs nemo-curator[text_cpu]; a GPU text path requires a CUDA 12 toolkit on Linux x86_64 with roughly 16 GB of GPU memory and a tested override file. Audio and video workflows that need FFmpeg are recommended to use a locally built container image. Slurm multi-node deployment is documented.

## Current Status

The README lists release 26.04 (Cosmos-Xenna 0.2.0 upgrade, simplified Resources API, Ray runtime upgrade) and 26.02 (Ray-based architecture for all modalities). It notes that from 26.09 NVIDIA no longer publishes new Curator container images on NGC, while existing images remain downloadable and GitHub source and releases continue.

## Features
- Text deduplication (exact, fuzzy, substring), classification, quality filtering, and language detection
- Image aesthetic and NSFW filtering, embedding generation, and deduplication
- Video scene detection, clip extraction, motion filtering, and deduplication
- Audio ASR transcription, quality assessment, WER filtering, and speaker diarization
- Stage-based pipelines with per-stage resource declarations and auto-scaling executors
- GPU-accelerated and distributed execution on Ray clusters
- Slurm multi-node deployment
- OpenAI-compatible inference server for synthetic data generation
- Nemotron-CC curation and SDG recipes

## Integrations
Ray, NVIDIA RAPIDS, Cosmos-Xenna, Hugging Face, vLLM, Slurm, Docker, Kubernetes, Common Crawl, WebDataset

## Platforms
LINUX, API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
v1.4.0

## Links
- Website: https://github.com/NVIDIA-NeMo/Curator
- Documentation: https://docs.nvidia.com/nemo/curator/latest
- Repository: https://github.com/NVIDIA-NeMo/Curator
- EveryDev.ai: https://www.everydev.ai/tools/nvidia-nemo-curator