# AI Trains AI

> An open-source pipeline that uses reinforcement learning to train an AI agent that itself writes and submits RL training jobs for smaller AI models on real GPUs.

AI Trains AI is an open-source research project by Dan Austin that implements a nested reinforcement learning system: an outer RL loop trains a large agent (Qwen3.6-35B-A3B with LoRA), and that agent's action is to write complete RL training jobs for smaller models, which are then executed on real GPU hardware. The entire codebase, trained model weights, reward code, task families, and retrospective write-ups are publicly available under the MIT License.

## What It Is

AI Trains AI is an agent harness and RL training pipeline built around a two-loop architecture. The outer loop trains a "trainer agent" using GRPO via Thinking Machines' Tinker platform. The inner loop executes the jobs that agent writes — running GRPO training on small base models (Qwen3-0.6B and 1.7B) on Runpod GPU pods using the prime-rl framework. The reward signal that trains the outer agent comes from how much the inner models improve on a hidden evaluation set, creating a recursive structure: an RL loop whose action is to run another RL loop.

## How the Two-Loop Architecture Works

The system separates training into two entirely distinct stacks:

- **Outer loop**: Trains the trainer agent (Qwen3.6-35B-A3B, LoRA rank 8) using Tinker's managed RL API with importance-sampling GRPO. Each episode, the agent is given a task spec and uses file-editing tools to write a complete prime-rl training job, then submits it.
- **Inner loop**: A warm pool of up to 16 Runpod GPU pods picks up validated jobs from a file-backed queue, runs GRPO training on a small base model, and scores the checkpoint pre/post on a hidden eval. The uplift score flows back as the outer loop's episode reward.

Each outer-loop batch spawns approximately 40 real inner training jobs. An async off-policy setting (`max_steps_off_policy=2`) prevents slow episodes from blocking entire batches.

## Results and Generalization

Over 54 outer-loop training steps across three pilot runs (pilot-7, 7b, 7c), the project reports reward climbing from approximately 0.0 to a peak of ~0.63. The README describes two distinct learning "rungs": first, the agent learned process reliability (converting validation failures into completed episodes); second, it learned to write jobs that actually improved the inner models' scores. Notably, the trained agent was evaluated on a held-out task family ("triage") it never trained on, and the mean reward on that family rose from 0.399 (base model) to 0.545 at the step-34 checkpoint, suggesting some degree of skill transfer. The agent also learned to prefer the stronger 1.7B base model over the 0.6B model, with its selection rate shifting from 42% to 95% after the `get_baseline_scores` tool was introduced.

## Infrastructure and Tooling

The orchestration runs on a CPU box rented via Nebius. Key components include:

- **Tinker** (Thinking Machines): managed RL API for outer-loop agent training
- **prime-rl** (PrimeIntellect): GRPO training framework for inner-loop jobs
- **Verifiers** (PrimeIntellect): environment and rubric framework used by agent-written jobs
- **Runpod**: GPU pod fleet (up to 16 concurrent pods; A40, RTX 4090, RTX A6000, RTX A5000)
- **vLLM**: used for pre/post checkpoint scoring on the hidden eval

The GPU selection is data-driven: a benchmark matrix over GPU × base-model combinations found 2× RTX A5000 to be the cost winner, though availability meant the headline arc ran mostly on A40s.

## Update: v0.1 Release (July 2026)

The repository was created on July 7, 2026, and version v0.1 was published on July 14, 2026. The trained LoRA adapter (rank 8, ~560MB) from the step-34 checkpoint is released on Hugging Face under Apache-2.0 as `Danau5tin/ai-trains-ai-trainer`. The README includes full retrospective write-ups of every pilot run, including failures, and the repository has 114 stars and 8 forks as of mid-July 2026. The project is described by its author as a personal research project, with future directions including iterative multi-job experimentation and richer stateful task environments.

## Features
- Nested RL training pipeline (outer agent trains inner models)
- Agent harness with file-editing and job-submission tools
- Warm GPU pod fleet orchestration via Runpod
- GRPO-based outer-loop training via Tinker
- prime-rl inner-loop training with verifiers environments
- File-backed job queue (queued → running → done)
- Async off-policy training to eliminate straggler bottlenecks
- Hidden eval scoring pre/post checkpoint with vLLM
- Reward design combining validation efficiency, job quality, and train speed
- Six task families including a held-out generalization probe
- Published LoRA adapter weights on Hugging Face
- Full cost metering per episode and per batch
- Retrospective write-ups of every pilot including failures
- GPU benchmark matrix for cost-optimal pod selection

## Integrations
Tinker (Thinking Machines), prime-rl (PrimeIntellect), Verifiers (PrimeIntellect), Runpod, Nebius, Hugging Face, vLLM, OpenRouter, Qwen3.6-35B-A3B, Qwen3-0.6B, Qwen3-1.7B, PEFT, uv

## Platforms
CLI, API

## Pricing
Open Source

## Version
v0.1

## Links
- Website: https://github.com/Danau5tin/ai-trains-ai
- Documentation: https://github.com/Danau5tin/ai-trains-ai/blob/main/docs/architecture.md
- Repository: https://github.com/Danau5tin/ai-trains-ai
- EveryDev.ai: https://www.everydev.ai/tools/ai-trains-ai