Harbor
Open-source framework for evaluating and optimizing agents and language models in sandboxed container environments.
At a Glance
About Harbor
Harbor is an open-source framework from the creators of Terminal-Bench for evaluating and optimizing agents and language models. It lets users run agents against benchmark datasets in sandboxed environments, locally with Docker or in parallel on cloud sandbox providers. It is installed as a Python package and driven from a command line.
What It Is
Harbor is an agent evaluation harness. Users specify sandboxed agent tasks, then run an agent and model against a dataset to measure performance. The README describes it as the official harness for Terminal-Bench-2.0, and it can also run third-party benchmarks such as SWE-Bench and Aider Polyglot.
How the Workflow Runs
A run is launched with the harbor run command, choosing a dataset and version, an agent, and a model. By default the benchmark runs locally in Docker. Passing an environment flag moves execution to a cloud provider such as Daytona, with a concurrency setting controlling how many trials run in parallel. harbor datasets list shows the supported benchmarks, and harbor run --help shows supported agents and options.
Agents, Benchmarks and Rollouts
The README says Harbor can evaluate arbitrary agents such as Claude Code, OpenHands and Codex CLI. Users can build and share their own benchmarks and environments. Harbor can also generate rollouts for RL optimization. Parallel experiments across thousands of environments are supported through providers including Daytona, Modal, LangSmith, Blaxel, Novita Sandbox, Tensorlake, Runta and Sail.
Setup Path
Install with uv tool install harbor or pip install harbor. A separate Harbor Cookbook provides end-to-end examples and guides. The project is licensed under Apache 2.0.
Community Discussions
Be the first to start a conversation about Harbor
Share your experience with Harbor, ask questions, or help others learn from your insights.
Pricing
Open Source
Harbor core framework, self-hosted and licensed under Apache License 2.0.
- Apache License 2.0 (Apache-2.0)
- Install via uv tool install harbor or pip install harbor
- Evaluate agents such as Claude Code, OpenHands, Codex CLI
- Build and share your own benchmarks and environments
- Run locally with Docker or on cloud providers like Daytona and Modal
Capabilities
Key Features
- Evaluate arbitrary agents such as Claude Code, OpenHands and Codex CLI
- Build and share custom benchmarks and environments
- Run thousands of sandboxed environments in parallel via cloud providers
- Generate rollouts for RL optimization
- Official harness for Terminal-Bench-2.0
- Run third-party benchmarks such as SWE-Bench and Aider Polyglot
- Local Docker execution or cloud execution via an environment flag
- Dataset listing and versioned dataset runs from the CLI
