TurboGPT

TurboGPT is an open-source project hosted on GitHub under the user lostmsu. Its README describes it as tiny byte-level GPT training in CUDA C++, released under the MIT license. The repository lists a v0.0.1 release published on 2026-09-29.

What It Is

TurboGPT is a training program for small GPT-style language models that operates on raw bytes. It is written in C++ with CUDA and, per the repository description, targets training a tiny GPT in under a minute on CUDA-capable GPUs only. It is a command-line program rather than a hosted service.

Build and Run Path

The README documents two build routes. On Linux or NixOS, the project is built with nix-build. On Windows, it uses Visual Studio 2022 C++ tools and CUDA 13.4 through a build.ps1 script that takes a CudaArch value, the GPU compute capability from NVIDIA's CUDA GPU list.

A run is started with the turbogpt executable, pointing it at a data file (the example uses hn1g.txt) and a log directory.

Checkpoints and Logging

Each run stores a checkpoint containing model, optimizer, scheduler, and trainer state, and the --load option resumes from it. The README states that logs are TensorBoard-compatible, with one report per batch, capped at 8Mi reports, and flushed with periodic or final checkpoints. A report.json file is derived from the log directory.

Reported Result

The README reports 2.5295 bits per byte (BPB) on the hn1g dataset after 1.5G training tokens. This is the author's own figure, not an independent benchmark.

Licensing and Provenance

The license file lists copyright for turboGPT contributors and for Andrej Karpathy (minGPT), indicating the project acknowledges minGPT in its license notice. The repository includes a Python verification script, tests/verify.py, for testing.