# Gigatoken

> An open-source Python library that tokenizes language model text data at GB/s speeds, claiming up to ~1000x faster throughput than HuggingFace tokenizers, with drop-in compatibility.

Gigatoken is an open-source tokenization library for language models, written primarily in Rust and published under the MIT License by Marcel Rød. It installs via `pip install gigatoken` and offers both a native API and drop-in compatibility modes for HuggingFace Tokenizers and tiktoken. The project has accumulated over 3,400 GitHub stars since its creation in November 2025.

## What It Is

Gigatoken is a high-performance byte-pair encoding (BPE) tokenizer designed to replace the tokenization step in LLM training and data preprocessing pipelines. It targets the CPU-bound bottleneck of pretokenization — the regex-heavy step that splits raw text before BPE merges — and replaces it with a custom SIMD-accelerated implementation. The result is a library that processes text at gigabytes per second rather than megabytes per second, while producing outputs that match HuggingFace Tokenizers exactly in compatibility mode.

## How the Speed Is Achieved

The README describes three main sources of speedup:

- **SIMD pretokenization**: The regex-based pretokenizer used by nearly all tokenizers is replaced with a hand-optimized SIMD implementation (AVX-512, AVX2, and NEON), eliminating the regex engine overhead entirely.
- **Pretoken caching**: A heavily optimized cache maps previously seen words to their encoded tokens. Because pretoken distributions are long-tailed, cache hit rates are high, and the cache hierarchy is tuned to minimize lookup cost as the cache grows.
- **Minimized Python and thread overhead**: The Rust implementation reads file data directly, avoids unnecessary Python object creation, and uses concurrent data structures to maximize parallelism without inter-thread communication.

## Benchmark Results

The project publishes detailed throughput numbers across three CPU platforms on an 11.9 GB OpenWebText file. On a dual-socket AMD EPYC 9565 (144 cores), Gigatoken encodes GPT-2 tokens at 24.53 GB/s versus 24.8 MB/s for HuggingFace tokenizers — a 989× speedup. On an Apple M4 Max (16 cores), the same comparison yields 8.79 GB/s versus 6.9 MB/s (1,268×). On an AMD Ryzen 7 9800X3D (16 cores), GPT-2 reaches 6.27 GB/s versus 59.0 MB/s (106×). The library supports a wide range of modern tokenizers including GPT-2, Llama 3/3.1/3.2/3.3/4, Qwen 2/3, DeepSeek V3/R1, Phi-4, Gemma, GLM, Kimi K2, and more.

## Supported Tokenizers and Compatibility

Gigatoken accepts HuggingFace model names directly (e.g., `gt.Tokenizer("Qwen/Qwen3-8B")`) and wraps existing HuggingFace or tiktoken tokenizer objects. In compatibility mode, outputs are validated to match HuggingFace Tokenizers exactly. The native `encode_files` API allows the Rust layer to read files directly for maximum throughput. Known limitations include no WordPiece support, limited optimization for SentencePiece-based tokenizers (used by Gemma and BERT-style models), and limited Windows testing (WSL recommended).

## Current Status and Known Issues

The repository was created in November 2025 and was last pushed in July 2026, indicating active development. The project is tagged with topics: llm, nlp, tokenization, tokenizer. The author notes several open issues and planned improvements, including Python version-specific ABI specialization (expected to yield ~2× additional speedup for overhead-bound cases), file sink support in the native API, and broader WordPiece support. SentencePiece optimization is explicitly marked as low priority. The project can be tested without installation using `uvx gigatoken bench` against any HuggingFace model repo.

## Features
- Up to ~1000x faster tokenization than HuggingFace tokenizers
- Drop-in compatibility with HuggingFace Tokenizers and tiktoken APIs
- SIMD-accelerated pretokenization (AVX-512, AVX2, NEON)
- Pretoken caching for long-tailed token distributions
- Native file-reading API for maximum throughput
- Supports GPT-2, Llama 3/4, Qwen 2/3, DeepSeek, Phi-4, Gemma, GLM, Kimi K2, and more
- Multithreaded Rust implementation with minimal Python overhead
- CLI benchmarking tool via uvx
- Exact output matching with HuggingFace Tokenizers in compatibility mode
- Supports modern x86 and ARM CPUs

## Integrations
HuggingFace Tokenizers, tiktoken, Python, Rust, pip, uvx

## Platforms
WINDOWS, MACOS, API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Links
- Website: https://github.com/marcelroed/gigatoken
- Documentation: https://github.com/marcelroed/gigatoken#readme
- Repository: https://github.com/marcelroed/gigatoken
- EveryDev.ai: https://www.everydev.ai/tools/gigatoken