# SIA (Self-Improving AI)

> Open-source framework that uses Meta, Target and Feedback agents to autonomously improve an AI agent's performance on a benchmark task.

SIA (Self-Improving AI) is an open-source framework from Hexo Labs that autonomously improves the performance of an AI system, a model or an agent, on a benchmark task. It is the official implementation of the paper "SIA: Self Improving AI with Harness & Weight Updates", and it is installed as a Python package with a command-line interface.

## What It Is

SIA is a self-improvement loop for AI agents. A user supplies a task, either one of the bundled tasks or a custom task directory, and SIA runs successive generations in which the agent is rewritten and re-evaluated. Per the paper abstract in the README, the loop updates both the harness and the weights of a task-specific agent.

## How the Loop Works

Three agent roles coordinate the process:

- The Meta-Agent reads the task description and generates an initial Target Agent tailored to the task.
- The Target Agent attempts the task and records its actions and results.
- The Feedback/Improvement Agent reviews the Target Agent's logs, identifies improvements, and updates the Target Agent.

After each generation the orchestrator runs the task's evaluator against held-out data and feeds the metrics into the next feedback prompt. Artifacts for each generation, such as the target agent code, execution logs and an improvement rationale, are written to a runs directory.

## Running and Configuring It

The `sia run` command executes the loop, and `sia web` serves a dashboard that visualizes runs, including per-generation code, prompts, scores and trajectories. Four built-in tasks ship with the package: gpqa, lawbench, longcot-chess and spaceship-titanic. Users can also bring their own task directory or bootstrap one from an MLE-Bench competition. Agent implementations include Claude Agent SDK and OpenHands, and JSON provider and profile files select models and endpoints. A Docker sandbox option isolates agent-generated code, which otherwise runs with host access by default.

## Reported Results

The README reports a 56.6% gain on LawBench, a 91.9% runtime reduction on GPU kernels, and a 502% improvement on single-cell RNA denoising over baseline. These are figures from the authors' paper.

## Features
- Meta, Target and Feedback agent loop across generations
- Updates both agent harness and weights
- Four built-in tasks: gpqa, lawbench, longcot-chess, spaceship-titanic
- Bring-your-own task directories with custom evaluate.py evaluators
- MLE-Bench competition bootstrapping
- Live web dashboard for visualizing runs
- JSON provider and profile configuration for models
- Optional Docker sandbox for isolated execution
- Multiple agent implementations (Claude, OpenHands, pydantic-ai)

## Integrations
Anthropic, OpenAI, Google Gemini, OpenRouter, OpenHands, Claude Agent SDK, MLE-Bench, Kaggle API, Docker, Nebius

## Platforms
WINDOWS, MACOS, LINUX, ANDROID, IOS, WEB, API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Links
- Website: https://hexolabs.com/
- Documentation: https://github.com/hexo-ai/sia/blob/main/docs/configuration.md
- Repository: https://github.com/hexo-ai/sia
- EveryDev.ai: https://www.everydev.ai/tools/sia