# Synthetic Data Kit

> A Python CLI toolkit for generating synthetic QA, chain-of-thought and summary datasets to fine-tune LLMs.

Synthetic Data Kit is an open-source Python command-line toolkit published under the meta-llama GitHub organization. It turns documents into synthetic datasets for fine-tuning LLMs through a four-command pipeline, and the PyPI listing shows an alpha development status.

## What It Is

The kit prepares training data for fine-tuning, particularly for the Llama model family. It uses an LLM backend, either a vLLM server or an external API endpoint, to generate examples from source documents. It then converts them into formats that fine-tuning tools accept. It is aimed at developers and researchers whose raw data is not already structured as user/assistant conversations.

## The four-stage workflow

- `ingest` parses PDF, HTML, DOCX, PPTX, TXT files and YouTube transcripts. A `--multimodal` option extracts text and images from PDFs into a Lance dataset.
- `create` generates QA pairs, QA pairs with chain-of-thought reasoning, summaries, or multimodal QA pairs.
- `curate` uses Llama as a judge to filter examples against a quality threshold.
- `save-as` exports to formats such as Alpaca, OpenAI fine-tuning, ChatML or JSONL, with optional Hugging Face dataset storage.

A `system-check` command verifies that the backend is running.

## Configuration and control

Behavior is set through a YAML config file or CLI flags. Settings include the LLM provider, model, temperature, number of pairs, curation threshold and batch size. Documents of 8000 characters or more are split into overlapping chunks, with configurable size and overlap. Prompt templates can be customized, for example to target a legal-assistant dataset. Single files and whole directories are supported, and `--preview` shows which files would be processed.

## Features
- Four-command pipeline: ingest, create, curate, save-as
- QA pair generation
- Chain-of-thought reasoning trace generation
- Summary generation
- LLM-as-judge quality curation with threshold
- Multimodal ingestion and QA generation
- Parses PDF, HTML, DOCX, PPTX, TXT and YouTube transcripts
- Export to Alpaca, OpenAI fine-tuning, ChatML and JSONL formats
- Hugging Face dataset storage option
- Automatic document chunking with configurable size and overlap
- Batch directory processing and preview mode
- YAML config and custom prompt templates

## Integrations
vLLM, Llama API, Hugging Face, Lance, OpenAI fine-tuning format

## Platforms
WEB, API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
0.0.5

## Links
- Website: https://pypi.org/project/synthetic-data-kit/
- Documentation: https://github.com/meta-llama/synthetic-data-kit#readme
- Repository: https://github.com/meta-llama/synthetic-data-kit
- EveryDev.ai: https://www.everydev.ai/tools/synthetic-data-kit