Kiln Harness Tuner
An open-source CLI that runs an agentic optimization loop to tune an agent's tools, prompts, skills, model and settings against Kiln evals.
At a Glance
About Kiln Harness Tuner
Kiln Harness Tuner is an open-source command-line tool from Kiln-AI that optimizes an existing agent built as a task in Kiln. It plans experiments, measures them on train data, promotes candidates on validation data, and never reads the test split. It runs on a coding agent the user already has, such as Claude Code, Codex or OpenCode.
What It Is
Harness Tuner tunes everything around the model rather than only the prompt. The README lists tools, prompts and skills, model and settings (thinking level, temperature and others), and RAG, fine-tunes and sub-agents as tunable parts. It tries different approaches, not just different wording, such as one large prompt versus skills, or many small tools versus one general tool.
How the Tuning Loop Works
The user sets a goal of quality, cost or speed and can steer it in plain English. Work proceeds in waves until the budget or time limit runs out, then it wraps up and writes a report. Key mechanics described in the README:
- Each experiment runs only the evals it targets, and candidates are checked against every eval before promotion.
- Experiments form a graph that branches from earlier versions, and winners merge back.
- Train, val and test splits are enforced in code. Agents see only train, val decides promotions, and test is left for the final report.
- It surfaces several winners, such as best quality, cheapest at good quality, balanced and fastest, which can be compared in Kiln.
- Tool rewrites are done as wrapper code tools, leaving the original tools untouched.
The README reports that on one production multi-turn agent with over 200 tools, tuning gave 2.5x fewer failures, 6x lower cost and 2x lower latency on held-out data; this is a vendor-published result.
Setup Path
It installs with uv and is launched with the harness-tuner command, which walks through starting Kiln and picking the project and task. It requires Kiln on the same machine, an agent task with evals, a supported coding agent, and model provider keys stored in Kiln. A demo mode (harness-tuner --demo) tunes a bundled forum-moderation task. The tool itself is free, but runs require LLM API calls.
Works with Kiln and Seahaven
Kiln provides evals, experiment tracking, a Compare view and an Experiment Graph view. Seahaven provides synthetic worlds for agents that change data, and synthetic users support multi-turn evals.
Community Discussions
Be the first to start a conversation about Kiln Harness Tuner
Share your experience with Kiln Harness Tuner, ask questions, or help others learn from your insights.
Pricing
Open Source
Harness Tuner is free and open source under the MIT license; LLM API calls are an external cost.
- MIT licensed
- Runs on Claude Code, Codex or OpenCode using your existing subscription
- Requires LLM API calls (external cost); demo project run costs about $2 via OpenRouter
- Works with the free Kiln app
Capabilities
Key Features
- Tunes tools, prompts, skills, model and hyperparameters
- Rewrites tools as wrapper code tools
- Optimizes for quality, cost or speed
- Multi-eval tracking with targeted experiments
- Experiment graph with branching and merging
- Train/val/test splits enforced in code
- Multiple winning candidates to compare
- Runs on Claude Code, Codex or OpenCode
- Demo project mode
- Writes a report after each run
Integrations
Demo Video

