Synthetic Data Kit

Synthetic Data Kit is an open-source Python command-line toolkit published under the meta-llama GitHub organization. It turns documents into synthetic datasets for fine-tuning LLMs through a four-command pipeline, and the PyPI listing shows an alpha development status.

What It Is

The kit prepares training data for fine-tuning, particularly for the Llama model family. It uses an LLM backend, either a vLLM server or an external API endpoint, to generate examples from source documents. It then converts them into formats that fine-tuning tools accept. It is aimed at developers and researchers whose raw data is not already structured as user/assistant conversations.

The four-stage workflow

• ingest parses PDF, HTML, DOCX, PPTX, TXT files and YouTube transcripts. A --multimodal option extracts text and images from PDFs into a Lance dataset.
• create generates QA pairs, QA pairs with chain-of-thought reasoning, summaries, or multimodal QA pairs.
• curate uses Llama as a judge to filter examples against a quality threshold.
• save-as exports to formats such as Alpaca, OpenAI fine-tuning, ChatML or JSONL, with optional Hugging Face dataset storage.

A system-check command verifies that the backend is running.

Configuration and control

Behavior is set through a YAML config file or CLI flags. Settings include the LLM provider, model, temperature, number of pairs, curation threshold and batch size. Documents of 8000 characters or more are split into overlapping chunks, with configurable size and overlap. Prompt templates can be customized, for example to target a legal-assistant dataset. Single files and whole directories are supported, and --preview shows which files would be processed.