EveryDev.ai
Subscribe
Home
Tools

5,901+ AI tools

  • Featured
  • Gems
  • Trending
  • New
  • Rate tools
  • Compare
  • Arena
Categories
  • Agents2972
  • Coding2248
  • Infrastructure1067
  • Projects818
  • MCP794
  • Research744
  • Analytics565
  • Marketing527
  • Design488
  • Testing476
  • Data442
  • Security412
  • Integration288
  • Communication284
  • Prompts283
  • Extensions238
  • Voice213
  • DevOps190
  • Commerce183
  • Web128
  • Learning123
  • Finance44
AI Tools by Topic
  • AI Coding Assistants
  • Agent Frameworks
  • MCP Servers
  • AI Prompt Tools
  • Vibe Coding Tools
  • AI Design Tools
  • AI Database Tools
  • AI Website Builders
  • AI Testing Tools
  • LLM Evaluations
Follow Us
  • X / Twitter
  • LinkedIn
  • Reddit
  • Discord
  • Threads
  • Bluesky
  • Mastodon
  • YouTube
  • GitHub
  • Instagram
Get Started
  • Users
  • Rate Tools
  • About
  • Editorial Standards
  • Corrections & Disclosures
  • Community Guidelines
  • Advertise
  • Contact Us
  • Newsletter
  • Submit a Tool
  • Start a Discussion
  • Write A Blog
  • Share A Build
  • Terms of Service
  • Privacy Policy
Explore with AI
  • ChatGPT
  • Gemini
  • Claude
  • Grok
  • Perplexity
Agent Experience
  • llms.txt
Theme
With AI, Everyone is a Dev. EveryDev.ai ©
    1. Home
    2. Tools
    3. Kiln Harness Tuner
    Kiln Harness Tuner icon

    Kiln Harness Tuner

    LLM Evaluations

    An open-source CLI that runs an agentic optimization loop to tune an agent's tools, prompts, skills, model and settings against Kiln evals.

    Visit Website

    At a Glance

    Pricing
    Open Source

    Harness Tuner is free and open source under the MIT license; LLM API calls are an external cost.

    Engagement

    Available On

    API
    CLI

    Resources

    WebsiteDocsGitHubllms.txt

    Topics

    LLM EvaluationsCommand Line AssistantsAgent Harness

    Alternatives

    GambitYoloAITerminal-Bench-Science
    Developer
    Chesterfield Laboratories IncToronto, CanadaEst. 2024

    Listed Nov 2026

    About Kiln Harness Tuner

    Kiln Harness Tuner is an open-source command-line tool from Kiln-AI that optimizes an existing agent built as a task in Kiln. It plans experiments, measures them on train data, promotes candidates on validation data, and never reads the test split. It runs on a coding agent the user already has, such as Claude Code, Codex or OpenCode.

    What It Is

    Harness Tuner tunes everything around the model rather than only the prompt. The README lists tools, prompts and skills, model and settings (thinking level, temperature and others), and RAG, fine-tunes and sub-agents as tunable parts. It tries different approaches, not just different wording, such as one large prompt versus skills, or many small tools versus one general tool.

    How the Tuning Loop Works

    The user sets a goal of quality, cost or speed and can steer it in plain English. Work proceeds in waves until the budget or time limit runs out, then it wraps up and writes a report. Key mechanics described in the README:

    • Each experiment runs only the evals it targets, and candidates are checked against every eval before promotion.
    • Experiments form a graph that branches from earlier versions, and winners merge back.
    • Train, val and test splits are enforced in code. Agents see only train, val decides promotions, and test is left for the final report.
    • It surfaces several winners, such as best quality, cheapest at good quality, balanced and fastest, which can be compared in Kiln.
    • Tool rewrites are done as wrapper code tools, leaving the original tools untouched.

    The README reports that on one production multi-turn agent with over 200 tools, tuning gave 2.5x fewer failures, 6x lower cost and 2x lower latency on held-out data; this is a vendor-published result.

    Setup Path

    It installs with uv and is launched with the harness-tuner command, which walks through starting Kiln and picking the project and task. It requires Kiln on the same machine, an agent task with evals, a supported coding agent, and model provider keys stored in Kiln. A demo mode (harness-tuner --demo) tunes a bundled forum-moderation task. The tool itself is free, but runs require LLM API calls.

    Works with Kiln and Seahaven

    Kiln provides evals, experiment tracking, a Compare view and an Experiment Graph view. Seahaven provides synthetic worlds for agents that change data, and synthetic users support multi-turn evals.

    Kiln Harness Tuner - 1

    Community Discussions

    Be the first to start a conversation about Kiln Harness Tuner

    Share your experience with Kiln Harness Tuner, ask questions, or help others learn from your insights.

    Pricing

    OPEN SOURCE

    Open Source

    Harness Tuner is free and open source under the MIT license; LLM API calls are an external cost.

    • MIT licensed
    • Runs on Claude Code, Codex or OpenCode using your existing subscription
    • Requires LLM API calls (external cost); demo project run costs about $2 via OpenRouter
    • Works with the free Kiln app

    Capabilities

    Key Features

    • Tunes tools, prompts, skills, model and hyperparameters
    • Rewrites tools as wrapper code tools
    • Optimizes for quality, cost or speed
    • Multi-eval tracking with targeted experiments
    • Experiment graph with branching and merging
    • Train/val/test splits enforced in code
    • Multiple winning candidates to compare
    • Runs on Claude Code, Codex or OpenCode
    • Demo project mode
    • Writes a report after each run

    Integrations

    Kiln
    Seahaven
    Claude Code
    Codex
    OpenCode
    OpenRouter
    uv
    API Available
    View Docs

    Demo Video

    Kiln Harness Tuner Demo Video
    Watch on YouTube

    Ratings & Reviews

    No ratings yet

    Be the first to rate Kiln Harness Tuner and help others make informed decisions.

    Rate other tools you’ve used

    Developer

    Chesterfield Laboratories Inc

    Chesterfield Laboratories Inc develops Kiln AI, a development platform for teams shipping AI products. The company describes its founding team as five people from Apple and Microsoft. Steve Cosman is listed as Founder & CEO.

    Founded 2024
    Toronto, Canada
    5 employees

    Used by

    Vector Institute for AI Research…
    People from Apple, Alibaba, 3M,…
    Read more about Chesterfield Laboratories Inc
    WebsiteGitHub
    2 tools in directory

    Similar Tools

    Gambit icon

    Gambit

    Gambit is an open-source agent harness framework by Bolt Foundry for building, running, and verifying LLM workflows using typed decks.

    YoloAI icon

    YoloAI

    A lightweight, no-frills AI command-line tool hosted on GitHub that lets you interact with AI models directly from your terminal.

    Terminal-Bench-Science icon

    Terminal-Bench-Science

    An open-source benchmark for evaluating AI agents on expert-curated research workflows across life, physical, earth, mathematical, and engineering sciences.

    Browse all tools

    Related Topics

    LLM Evaluations

    Platforms and frameworks for evaluating, testing, and benchmarking LLM systems and AI applications. These tools provide evaluators and evaluation models to score AI outputs, measure hallucinations, assess RAG quality, detect failures, and optimize model performance. Features include automated testing with LLM-as-a-judge metrics, component-level evaluation with tracing, regression testing in CI/CD pipelines, custom evaluator creation, dataset curation, and real-time monitoring of production systems. Teams use these solutions to validate prompt effectiveness, compare models side-by-side, ensure answer correctness and relevance, identify bias and toxicity, prevent PII leakage, and continuously improve AI product quality through experiments, benchmarks, and performance analytics.

    246 tools

    Command Line Assistants

    AI-powered command-line assistants that help developers navigate, search, and execute terminal commands with intelligent suggestions and context awareness.

    423 tools

    Agent Harness

    Infrastructure, orchestrators, and task runners that wrap around LLM coding agents — covering session management, context delivery, worktree isolation, architecture enforcement, and issue-to-PR pipelines.

    490 tools
    Browse all topics
    Back to all toolsSuggest an edit
    ratings
    discussions