SIA (Self-Improving AI)

SIA (Self-Improving AI) is an open-source framework from Hexo Labs that autonomously improves the performance of an AI system, a model or an agent, on a benchmark task. It is the official implementation of the paper "SIA: Self Improving AI with Harness & Weight Updates", and it is installed as a Python package with a command-line interface.

What It Is

SIA is a self-improvement loop for AI agents. A user supplies a task, either one of the bundled tasks or a custom task directory, and SIA runs successive generations in which the agent is rewritten and re-evaluated. Per the paper abstract in the README, the loop updates both the harness and the weights of a task-specific agent.

How the Loop Works

Three agent roles coordinate the process:

• The Meta-Agent reads the task description and generates an initial Target Agent tailored to the task.
• The Target Agent attempts the task and records its actions and results.
• The Feedback/Improvement Agent reviews the Target Agent's logs, identifies improvements, and updates the Target Agent.

After each generation the orchestrator runs the task's evaluator against held-out data and feeds the metrics into the next feedback prompt. Artifacts for each generation, such as the target agent code, execution logs and an improvement rationale, are written to a runs directory.

Running and Configuring It

The sia run command executes the loop, and sia web serves a dashboard that visualizes runs, including per-generation code, prompts, scores and trajectories. Four built-in tasks ship with the package: gpqa, lawbench, longcot-chess and spaceship-titanic. Users can also bring their own task directory or bootstrap one from an MLE-Bench competition. Agent implementations include Claude Agent SDK and OpenHands, and JSON provider and profile files select models and endpoints. A Docker sandbox option isolates agent-generated code, which otherwise runs with host access by default.

Reported Results

The README reports a 56.6% gain on LawBench, a 91.9% runtime reduction on GPU kernels, and a 502% improvement on single-cell RNA denoising over baseline. These are figures from the authors' paper.