Gigatoken

Gigatoken is an open-source tokenization library for language models, written primarily in Rust and published under the MIT License by Marcel Rød. It installs via pip install gigatoken and offers both a native API and drop-in compatibility modes for HuggingFace Tokenizers and tiktoken. The project has accumulated over 3,400 GitHub stars since its creation in November 2025.

What It Is

Gigatoken is a high-performance byte-pair encoding (BPE) tokenizer designed to replace the tokenization step in LLM training and data preprocessing pipelines. It targets the CPU-bound bottleneck of pretokenization — the regex-heavy step that splits raw text before BPE merges — and replaces it with a custom SIMD-accelerated implementation. The result is a library that processes text at gigabytes per second rather than megabytes per second, while producing outputs that match HuggingFace Tokenizers exactly in compatibility mode.

How the Speed Is Achieved

The README describes three main sources of speedup:

• SIMD pretokenization: The regex-based pretokenizer used by nearly all tokenizers is replaced with a hand-optimized SIMD implementation (AVX-512, AVX2, and NEON), eliminating the regex engine overhead entirely.
• Pretoken caching: A heavily optimized cache maps previously seen words to their encoded tokens. Because pretoken distributions are long-tailed, cache hit rates are high, and the cache hierarchy is tuned to minimize lookup cost as the cache grows.
• Minimized Python and thread overhead: The Rust implementation reads file data directly, avoids unnecessary Python object creation, and uses concurrent data structures to maximize parallelism without inter-thread communication.

Benchmark Results

The project publishes detailed throughput numbers across three CPU platforms on an 11.9 GB OpenWebText file. On a dual-socket AMD EPYC 9565 (144 cores), Gigatoken encodes GPT-2 tokens at 24.53 GB/s versus 24.8 MB/s for HuggingFace tokenizers — a 989× speedup. On an Apple M4 Max (16 cores), the same comparison yields 8.79 GB/s versus 6.9 MB/s (1,268×). On an AMD Ryzen 7 9800X3D (16 cores), GPT-2 reaches 6.27 GB/s versus 59.0 MB/s (106×). The library supports a wide range of modern tokenizers including GPT-2, Llama 3/3.1/3.2/3.3/4, Qwen 2/3, DeepSeek V3/R1, Phi-4, Gemma, GLM, Kimi K2, and more.

Supported Tokenizers and Compatibility

Gigatoken accepts HuggingFace model names directly (e.g., gt.Tokenizer("Qwen/Qwen3-8B")) and wraps existing HuggingFace or tiktoken tokenizer objects. In compatibility mode, outputs are validated to match HuggingFace Tokenizers exactly. The native encode_files API allows the Rust layer to read files directly for maximum throughput. Known limitations include no WordPiece support, limited optimization for SentencePiece-based tokenizers (used by Gemma and BERT-style models), and limited Windows testing (WSL recommended).

Current Status and Known Issues

The repository was created in November 2025 and was last pushed in July 2026, indicating active development. The project is tagged with topics: llm, nlp, tokenization, tokenizer. The author notes several open issues and planned improvements, including Python version-specific ABI specialization (expected to yield ~2× additional speedup for overhead-bound cases), file sink support in the native API, and broader WordPiece support. SentencePiece optimization is explicitly marked as low priority. The project can be tested without installation using uvx gigatoken bench against any HuggingFace model repo.