Augmentoolkit

Augmentoolkit is an MIT-licensed project by Evan Armstrong that generates datasets from user-supplied documents and can train custom LLMs on them. Users upload documents through an interface or CLI, run dataset generation, and optionally continue to model training. The latest listed release is v3.0.0, published in June 2025.

What It Is

Augmentoolkit is a dataset-generation and fine-tuning workflow for building domain-expert language models. The README describes it as creating datasets that update a model's knowledge so it becomes an expert in a chosen area, such as research papers in a field or the lore of a fictional universe. It also produces a RAG-ready dataset and can start an inference server for traditional grounding.

Pipelines and Outputs

The project ships several pipelines that can be composed or run separately: complete factual generation, multi-source and single-source recall data, representation variation, correction data with loss-masked mistakes, RAG data, a traditional classifier bootstrapper, generic data rephrasing, an experimental GRPO pipeline, and RPToolkit for multi-turn roleplay data. Utilities include a basic LLM server, a RAG LLM server, and Discord bot hosting for trained models. New pipelines are written as Python functions, with templates and annotated configs provided.

Setup Path

Start scripts exist for macOS and Linux, with separate scripts for local inference, and Python 3.11 is required. Linux local inference supports a tensor-parallelism option for multiple GPUs. On Windows the README recommends WSL or the CLI. Dataset generation can run offline on local hardware or use an open-source LLM API, and training can be switched off with the do_train config option.

Training Considerations

The README notes that factual fine-tuning needs enough optimizer steps, so small datasets may need generation-count settings increased to scale up the data. It also documents turning off sample packing as an untested emergency option for very small datasets.