# TurboGPT

> Tiny byte-level GPT training in CUDA C++ that trains small models quickly on NVIDIA GPUs.

TurboGPT is an open-source project hosted on GitHub under the user lostmsu. Its README describes it as tiny byte-level GPT training in CUDA C++, released under the MIT license. The repository lists a v0.0.1 release published on 2026-09-29.

## What It Is

TurboGPT is a training program for small GPT-style language models that operates on raw bytes. It is written in C++ with CUDA and, per the repository description, targets training a tiny GPT in under a minute on CUDA-capable GPUs only. It is a command-line program rather than a hosted service.

## Build and Run Path

The README documents two build routes. On Linux or NixOS, the project is built with nix-build. On Windows, it uses Visual Studio 2022 C++ tools and CUDA 13.4 through a build.ps1 script that takes a CudaArch value, the GPU compute capability from NVIDIA's CUDA GPU list.

A run is started with the turbogpt executable, pointing it at a data file (the example uses hn1g.txt) and a log directory.

## Checkpoints and Logging

Each run stores a checkpoint containing model, optimizer, scheduler, and trainer state, and the --load option resumes from it. The README states that logs are TensorBoard-compatible, with one report per batch, capped at 8Mi reports, and flushed with periodic or final checkpoints. A report.json file is derived from the log directory.

## Reported Result

The README reports 2.5295 bits per byte (BPB) on the hn1g dataset after 1.5G training tokens. This is the author's own figure, not an independent benchmark.

## Licensing and Provenance

The license file lists copyright for turboGPT contributors and for Andrej Karpathy (minGPT), indicating the project acknowledges minGPT in its license notice. The repository includes a Python verification script, tests/verify.py, for testing.

## Features
- Byte-level GPT training
- CUDA C++ implementation
- Checkpoints with model, optimizer, scheduler and trainer state
- Resume training with --load
- TensorBoard-compatible logs
- report.json derived from log directory
- Nix build for Linux/NixOS
- Windows build script with CUDA architecture option
- Verification tests via Python script

## Integrations
TensorBoard, NVIDIA CUDA, Nix

## Platforms
WINDOWS, LINUX, API, CLI

## Pricing
Open Source

## Version
v0.0.1

## Links
- Website: https://github.com/lostmsu/TurboGPT
- Repository: https://github.com/lostmsu/TurboGPT
- EveryDev.ai: https://www.everydev.ai/tools/turbogpt