Distilabel

Distilabel is an open-source Python framework from Argilla for generating synthetic data and collecting AI feedback. It targets engineers who want programmatic, scalable pipelines based on published research methods. The latest release listed is 1.5.3, and the repository is licensed under Apache 2.0.

What It Is

Distilabel lets you define pipelines made of steps, tasks and LLMs to generate and judge data. It covers traditional NLP work such as classification and extraction, and LLM scenarios such as instruction following, dialogue generation and judging. Pipelines produce a Distiset dataset object that can be pushed to the Hugging Face Hub or exported to Argilla.

How the Workflow Operates

You install it with pip, optionally adding extras for specific LLM providers, then build a Pipeline. A basic example runs a TextGeneration task with an LLM over an input dataset and pushes the result to the Hub. The documentation also covers structured generation, offline batch generation, pipeline caching, resource assignment per step, and a CLI for exploring and re-running pipelines. Pipelines can be scaled and distributed with Ray.

Models and Integrations

The docs say it integrates AI feedback from LLM providers through a unified API. Extras cover Anthropic, Cohere, Groq, Hugging Face Inference Endpoints, transformers, LiteLLM, llama.cpp, Mistral AI, Ollama, OpenAI-compatible APIs, Vertex AI, vLLM and MLX. Outlines and Instructor support structured generation, and the docs also describe image generation models and tasks. Tutorials reproduce research methods such as UltraFeedback, DEITA, Prometheus 2 and APIGen.

Current Status

The README notes that the original authors have moved on to other projects. Community collaborators now maintain the project and are working toward the next release, with the develop branch holding the latest fixes.