Transformer Lab

Transformer Lab is an open-source machine learning research platform designed for frontier AI/ML workflows. It unifies environment management, compute coordination, and experiment tracking into a single workspace, replacing brittle bash scripts and scattered tooling. The platform supports local, on-premises, and cloud deployments, making it suitable for individual researchers and large ML teams alike.

• Distributed Training Orchestration: Run jobs locally, on-prem, or in the cloud without rewriting scripts or managing Slurm templates; scheduling and telemetry keep GPU usage visible.
• Advanced Training & Multimodal Workflows: Supports pre-training, fine-tuning, and evaluation of LLMs, Diffusion, and Audio models, with production-ready implementations of DPO, ORPO, SIMPO, and GRPO.
• Experiment Tracking: Automatically records hyperparameters, code versions, metrics, and logs for every run, enabling easy comparison and resumption of experiments.
• Artifact, Dataset & Checkpoint Management: Systematically tracks which code, dataset version, and config produced each checkpoint; synchronizes data across nodes even with ephemeral compute.
• RLHF & Preference Optimization: Complete RLHF pipeline with reward modeling handles orchestration automatically, from data processing to final model outputs.
• Comprehensive Evaluations: Run Eleuther Harness benchmarks, LLM-as-a-Judge comparisons, and objective metrics; red-team models and visualize results over time with exportable dashboards.
• Broad Integrations: Works with Weights & Biases, GitHub, SkyPilot, Slurm, Kubernetes, Ray, Hugging Face TRL, Unsloth, PyTorch, and MLX on NVIDIA, AMD, TPUs, or Apple Silicon.
• Open Source: Freely available on GitHub; get started by visiting the documentation and installing the app on your preferred compute environment.