Harbor and Harbor Cookbook: Grade Your Agents Inside Real Environments

⬅️ Back to Tools

Harbor and Harbor Cookbook: Grade Your Agents Inside Real Environments

What it isAn eval framework that runs agents inside Docker environments and scores what they did, plus a cookbook of runnable eval recipes and RL optimization examples
PlatformmacOS / Linux / Windows CLI (Python, via pip or uv); trial environments need local Docker or a cloud sandbox provider
PriceFree, open source (Apache-2.0); you pay for model API calls and compute
Linkgithub.com/harbor-framework/harbor · github.com/harbor-framework/harbor-cookbook

Most agent arguments I see are vibes. Someone likes Claude Code, someone else swears by Codex, and nobody has a number. Harbor is the Terminal-Bench team’s answer: stop grading chat transcripts and start grading work done inside real environments. I installed it (version 0.23.0, one pip command) and walked the whole CLI surface to check the claims below. Full disclosure up front: I did not burn money running a full benchmark, and neither should you before reading the limits section.

  1. Evals happen at environment level, which is the whole point. The agent gets a Docker container with files, tools, and permissions, does the task, and a verifier checks the end state. Prompt in and response out is not enough once agents touch real systems, and Harbor is built for the world after that line.
  2. One CLI drives whatever agent you already use. harbor run -d "<dataset@version>" -m "<model>" -a "<agent>" covers Claude Code, Codex CLI, OpenHands, and more. It is also the official harness for Terminal-Bench 2.0, so harbor run --dataset terminal-bench@2.0 --agent claude-code --model anthropic/claude-opus-4-1 reproduces a known leaderboard instead of your own private benchmark.
  3. The scale knob is one flag. Local Docker with --n-concurrent 4 for an afternoon experiment, then --env daytona --n-concurrent 100 for a hundred parallel trials in the cloud. Daytona, Modal, LangSmith, Blaxel, Novita Sandbox, Tensorlake, and Runta are all supported backends. That local to cloud path with no rewrite is my favorite thing about the design.
  4. The cookbook’s simple-task recipe shows the full anatomy in five files. instruction.md is one sentence (“Create a file called hello.txt with “Hello, world!” as the content”), task.toml sets 120-second agent and verifier timeouts with 1 CPU and 2 GB RAM, and environment/, tests/, and solution/ hold the Dockerfile (ubuntu:24.04), the pytest assertions, and a reference solution. Five files and you own an eval.
  5. multi-reward is where it gets interesting. Instead of one pass or fail, two verifiers score correctness and performance separately into reward.json, with a custom per-dimension metric wired through config.yaml. And harbor trials start -p <recipe> runs a single trial with no metrics attached, which is the cheap iteration loop you want while building.
  6. Borrowing beats building. harbor datasets list points at the hub registry, and publish, download, and sync move tasks and datasets around with digests. Third-party benchmarks like SWE-Bench and Aider Polyglot are already there. My take: download three evals close to your workload before writing your first task.toml.
  7. The same environments feed RL. Harbor generates rollouts for optimization, and the cookbook ships five worked paths: harbor-rl, gepa (harness optimization on MedAgentBench), tinker-rl, prime-rl, and sky-rl. Eval infra that doubles as training infra is rare, and it is the reason labs take this project seriously at 5.3k stars and 1.8k forks.
  8. The CLI covers the full loop, not just running. init scaffolds tasks, check grades task quality against a rubric, analyze digs through trajectories, view opens a web browser for them, and agent list shows what is installed. I verified every one of these exists in the installed 0.23.0 binary.
  9. Honest limits, and there are real ones. A full trial run spends real money on model tokens plus Docker or cloud compute, so budget before you scale -n. harbor datasets list currently just prints the hub URL rather than listing anything locally. The Windows computer-use recipe requires Daytona cloud, not your laptop. And this moves fast (1,635 commits), so pin your version and read the changelog before upgrading mid-experiment.

Install & first run 

pip install harbor
harbor --version

That printed 0.23.0 on my machine. Then set a model key and run the smallest cookbook recipe:

export ANTHROPIC_API_KEY=<YOUR-KEY>
harbor run -p harbor_cookbook/recipes/simple-task --agent claude-code --model anthropic/claude-sonnet-4-6

Harbor builds the ubuntu:24.04 container, the agent writes hello.txt, the tests assert it, and a reward is recorded. You need Docker running and a key with budget. Iterating on your own task is cheaper with a single trial (harbor trials start -p <your-task>) and harbor view to inspect what the agent actually did. Everything beyond this lives in the two READMEs, which are genuinely good.

Worth your time if: your agents touch real systems and you are tired of arguing about models and prompts with opinions instead of scores.

Related TMFNK Content

Crepi il lupo! 🐺