Harbor is an open-source framework for running and analyzing agent and model evaluations in reproducible task environments. It is the official harness for Terminal-Bench and also supports other benchmark datasets.

What does a Harbor task contain?

A task combines an instruction, an environment definition, resource and timeout settings, and a verifier. The verifier writes a numeric reward so trials can be compared and inspected.

Why run Harbor evaluations in Vercel Sandbox?

Vercel Sandbox gives each trial an isolated Firecracker microVM, supports parallel runs, and can enforce network rules outside the guest. It avoids forcing every evaluation through one local Docker host.

Can a single benchmark score identify the best coding agent?

No. A score depends on the dataset, agent harness, model, prompt, tool versions, environment, attempt count, and verifier. Use repeated runs and task-level analysis before drawing conclusions.

How should API credentials be provided to evaluation sandboxes?

Prefer firewall-level credential brokering or narrowly scoped temporary credentials. Vercel can inject credentials into matching outbound requests so the value does not enter the guest filesystem or environment.

Should production teams create their own Harbor dataset?

Yes, when public benchmarks do not represent their repositories and failure costs. A small dataset of real, sanitized tasks often reveals more about deployment readiness than a larger generic leaderboard.

Coding Agent Evals With Harbor and Vercel Sandbox

A coding-agent benchmark is a software system, not a trivia quiz. The score depends on the task instructions, filesystem, installed tools, network access, time limit, verifier, agent harness, model configuration, and number of attempts. Change one of those and you may be measuring a different system.

Harbor packages those moving parts into repeatable tasks and jobs. Terminal-Bench provides difficult terminal-based tasks. Vercel Sandbox can now run Harbor trials in isolated Firecracker microVMs, making it practical to execute several evaluations in parallel without turning a laptop into a warm pile of fans.

The setup is useful, but the command is the easy part. A credible evaluation needs controlled inputs, reproducible environments, careful secret handling, and task-level analysis. Otherwise the leaderboard number acquires more confidence than the experiment earned.

What Harbor Actually Evaluates

Harbor is an open-source framework for evaluating agents and models in containerized environments. It can run coding agents such as Codex CLI, Claude Code, OpenHands, Gemini CLI, and others against registered or local datasets.

A dataset contains tasks. Each task has four important parts:

An instruction describing the work
An environment containing the starting files and dependencies
A verifier that tests the final state
Configuration for resources, timeouts, and metadata

Harbor calls one execution of an agent on one task a trial. A job is a collection of trials, possibly spanning several tasks, agents, models, or attempts.

That vocabulary matters when reading results. "The model scored 62%" hides which agent wrapper drove it, how many trials ran, and whether failures came from reasoning, setup, tool use, or the verifier.

Why Terminal-Bench Is Useful and Incomplete

Terminal-Bench 2.0 evaluates whether an agent can complete work in a terminal environment. The tasks are manually reviewed and come with verifiers, giving the benchmark more substance than asking a model to explain code in a chat box.

Terminal tasks exercise planning, command use, file editing, dependency management, debugging, and persistence across several steps. Those are closer to real engineering work than isolated code completion.

They are still a sample. Your product may care about a monorepo, a particular deployment stack, database migrations, accessibility checks, or a private framework. A strong Terminal-Bench result does not prove that an agent can safely modify your billing service. It shows performance on the benchmark's task distribution under its published setup.

Use a public benchmark for a shared baseline. Add a private, sanitized dataset for the work your team actually needs.

Why the Execution Environment Changes the Result

An agent that spends eight minutes installing dependencies has less time to solve the task. An image with a warm package cache behaves differently from a clean image. A network outage can look like a reasoning failure. A reused filesystem can leak artifacts from an earlier trial.

Vercel's Harbor integration runs each trial in its own Firecracker microVM. Harbor 0.22.0 and later can select the provider with --env vercel. Independent microVMs make parallelism easier while reducing cross-trial contamination.

Vercel's Sandbox documentation describes each sandbox as having its own filesystem and network. The environment supports root access, package installation, snapshots, and management through a CLI or SDK.

Isolation improves the experiment. It does not make the result automatically reproducible. Pin the task, agent, model, environment image, dependencies, and relevant configuration.

Install Harbor and Run a First Trial

Use a dedicated environment for the harness. The Vercel integration announcement uses the harbor[vercel] extra.

uv tool install 'harbor[vercel]'
harbor --help
harbor dataset list

Authenticate Vercel in the environment where Harbor runs. Supply model credentials through your chosen provider or use Vercel AI Gateway for a consistent model endpoint.

export VERCEL_TOKEN="<set-locally>"
export AI_GATEWAY_API_KEY="<set-locally>"

harbor run \
  -d terminal-bench/terminal-bench-2 \
  --agent codex \
  --model vercel_ai_gateway/openai/gpt-5.6-sol \
  --env vercel \
  --n-concurrent 4

Do not commit those values or paste them into a benchmark task. Use placeholders in documentation and load real secrets from the execution environment.

Start with low concurrency. Four parallel trials are easier to debug than forty. Once one task completes reliably and the outputs land where expected, increase concurrency within API, compute, and budget limits.

Understand the Files Harbor Produces

Harbor stores a job result and a directory for each trial. The exact layout evolves with the project, but the evaluation documentation describes job-level configuration and results alongside trial-level records.

Keep the complete artifact alongside the aggregate reward. A failed trial should retain enough evidence to answer:

Did the environment start correctly?
Did the agent understand the instruction?
Which commands and tools did it use?
Did it modify the intended files?
Did the verifier run successfully?
Did a timeout, provider error, or network rule cause the failure?

A score without the trace is hard to improve. It also makes infrastructure failures indistinguishable from agent failures.

Compare Models Without Changing Everything Else

Vercel AI Gateway lets Harbor switch models through one endpoint. That is convenient for controlled comparisons, provided the rest of the job remains fixed.

harbor run \
  -d terminal-bench/terminal-bench-2 \
  --agent codex \
  --model vercel_ai_gateway/openai/gpt-5.6-sol \
  --env vercel \
  --n-concurrent 8

harbor run \
  -d terminal-bench/terminal-bench-2 \
  --agent codex \
  --model vercel_ai_gateway/anthropic/claude-fable-5 \
  --env vercel \
  --n-concurrent 8

The comparison now changes the model while keeping the agent harness and dataset constant. That is narrower than comparing a vendor's complete coding product against another vendor's complete product. Both experiments can be useful, but they answer different questions.

Record provider and model identifiers exactly. A friendly alias that moves to a newer snapshot can silently invalidate a month-over-month trend.

Repeat Trials and Report Uncertainty

Agent runs are stochastic. One attempt per task can reward a lucky plan or punish one bad tool call. Run several attempts for important comparisons and report the distribution.

For binary tasks, useful summaries include pass rate, pass@1, repeated success rate, and confidence intervals. For graded tasks, report the mean and spread, then inspect the tasks near decision boundaries.

Do not rerun only the failures and combine those results with the original successes. That creates a retry policy, which is a legitimate product behavior, but it is no longer a single-attempt benchmark. Label it honestly.

A practical report might say:

Dataset: internal-migrations@1.3
Tasks: 24
Attempts per task: 5
Agent: codex 0.18.2
Model: provider/model-snapshot
Environment: vercel-sandbox image abc123
Timeout: 30 minutes
Success: 91/120 trials
Task pass@1: 16/24

That is far more useful than "76% on our benchmark."

Control Network Access During Trials

Coding agents often need package registries, source hosts, or model APIs. An unrestricted sandbox can also reach endpoints unrelated to the task.

Vercel Sandbox supports network policies that are enforced outside the guest. Its security guidance recommends restricting egress to required domains and updating the policy before untrusted work runs.

A safe lifecycle is:

Start with access needed to install pinned dependencies.
Prepare the workspace.
Switch to a narrow allowlist or deny-all policy.
Run the agent.
Export results.
Destroy the sandbox.

Network rules are part of the benchmark configuration. If one model can browse documentation and another cannot, the score compares tool access as well as model behavior.

Keep Credentials Outside the Guest

Putting a valuable API key in the sandbox environment lets any process inside the guest read it. That includes the agent, downloaded packages, test scripts, and a malicious task fixture.

Vercel supports credential brokering at the network firewall. The sandbox makes an outbound request, and the firewall injects a credential only when the request matches the configured destination. The secret does not enter the sandbox filesystem or environment.

Use a separate, narrowly scoped credential for evaluation even with brokering. Restrict it by provider, project, budget, and lifetime. Assume task content is untrusted, especially when importing a community dataset.

The safer question is not "Can the agent print the key?" It is "What can any process in this trial cause the credential to do?"

Build a Small Internal Dataset

Public benchmarks help with external comparison. Deployment decisions need tasks drawn from your own failure modes.

A Harbor task uses a task.toml, instruction.md, environment definition, test script, and optional reference solution. The project's adapter documentation describes the format and expects the verifier to write a numeric reward under /logs/verifier/reward.txt.

version = "1.0"

[task]
name = "nandann/fix-canonical-regression"

[metadata]
difficulty = "medium"
category = "web-development"
tags = ["nextjs", "seo", "testing"]

[agent]
timeout_sec = 1800.0

[verifier]
timeout_sec = 120.0

[environment]
cpus = 2
memory_mb = 4096
storage_mb = 10240

The instruction should state the goal, constraints, expected output, and files in scope. It should not expose the hidden test or reference solution. The verifier should test outcomes rather than matching one expected patch.

Good internal tasks come from real incidents and review comments after removing credentials, personal data, and proprietary material that the evaluation does not need. Include straightforward tasks, ambiguous tasks that should trigger clarification, and unsafe requests the agent should refuse or escalate.

Design Verifiers That Reward the Outcome

A verifier defines success more strongly than the prose description does. If the task asks an agent to fix a security bug but the verifier checks only that one file changed, the benchmark rewards activity instead of correctness.

Layer checks from cheap to expensive. Start with file and syntax validation, then run focused tests, static analysis, and behavior checks. Use an integration test when the task is about interaction between components. Avoid an LLM judge when a deterministic test can answer the question.

For a migration task, the verifier might check that the old API is absent, compile the project, run targeted tests, and exercise one representative request. It should also detect forbidden shortcuts, such as deleting the failing test or replacing a function with a hard-coded fixture.

#!/usr/bin/env bash
set -euo pipefail

score=0

if ! rg -n "legacyClient" /workspace/src; then
  score=$(awk 'BEGIN { print 0.2 }')
fi

if npm --prefix /workspace run typecheck; then
  score=$(awk -v s="$score" 'BEGIN { print s + 0.3 }')
fi

if npm --prefix /workspace test -- --runInBand migration.spec.ts; then
  score=$(awk -v s="$score" 'BEGIN { print s + 0.5 }')
fi

printf '%s\n' "$score" > /logs/verifier/reward.txt

This sketch shows a graded reward, not a universal verifier. In a security-sensitive task, any critical failure may need to force the score to zero. Document the scoring rule and test it against the reference solution, partial solutions, and deliberately deceptive patches.

Keep the verifier outside the agent's writable workspace where the environment supports that boundary. Do not include hidden test answers in instruction.md or readable setup files.

Separate Agent Quality From Harness Quality

The agent is more than the model. It includes its system instruction, tool set, context policy, retry behavior, shell integration, patch strategy, and verification loop.

Run three types of comparison:

Same harness, different models. This isolates the model choice as much as practical.
Same model, different harness versions. This measures orchestration and tool changes.
Complete product against complete product. This answers which packaged system performs better for the selected tasks.

Do not combine those results into one table without labeling the comparison type. A model may look weak because one harness truncates command output or gives it a shorter timeout. Another may look strong because its harness automatically retries failed tests.

Version the agent package and record its default prompts. If a hosted product changes those defaults without a visible version, record the evaluation date and any available build identifier.

Control Cost Before Increasing Concurrency

Parallel microVMs remove a local bottleneck, which makes spending easier to accelerate. Estimate the upper bound before starting a job.

maximum trials = tasks × models × agents × attempts
maximum model cost = maximum trials × per-trial token budget
maximum compute time = maximum trials × task timeout

A dataset with 100 tasks, three models, two agents, and five attempts produces 3,000 trials. A 30-minute timeout creates a theoretical 1,500 compute-hour ceiling before setup and verifier time.

Set job-level limits and provider budgets. Start with a representative subset that includes easy, medium, difficult, and historically flaky tasks. Abort the run if environment startup failures or provider errors exceed a threshold. Paying for another hundred trials will not repair a broken image.

Use concurrency as an experimental parameter. High concurrency can trigger provider rate limits, registry throttling, or shared downstream constraints that do not appear in smaller runs. If results change with concurrency, investigate the infrastructure before attributing the difference to the model.

Make Runs Reproducible Months Later

Save enough configuration to rerun the job after the default images and model aliases have changed.

At minimum, preserve:

Harbor and agent versions
Dataset name, version, and content digest
Environment image digest
Model provider and immutable model identifier where available
Generation parameters and reasoning setting
Tool configuration and network policy
Per-task timeout and resource limits
Attempt count, random seed where supported, and concurrency
Verifier version and result artifacts

Archive dependency lockfiles with private datasets. A floating apt-get install or unpinned npm dependency can turn a stable benchmark into a historical reenactment.

Snapshotting can reduce setup time, but label the snapshot and describe how it was created. A warm snapshot and a clean build are different environments. Periodically rebuild from the declared source to ensure the snapshot has not become the only surviving specification.

Add Evaluation Gates Without Worshipping the Score

An evaluation can run on a schedule, before a model change, after a harness upgrade, or before expanding an agent's permissions. Do not require the aggregate score to increase on every change. Sampling noise and task composition make that brittle.

Use several gates:

No regression on critical safety tasks
Aggregate success remains within an agreed tolerance
Cost and latency stay within task-specific budgets
Infrastructure failure rate remains below a threshold
No new destructive or policy-violating behavior

Keep a small "must pass" suite for actions with real consequences. A deployment agent that succeeds on 90% of refactors but occasionally promotes the wrong artifact is not ready for that authority.

When a gate fails, inspect the changed tasks and traces. A lower aggregate score caused by two new difficult tasks is different from an existing task failing after a tool update.

Avoid Common Benchmark Traps

Contamination is the obvious concern. If tasks and solutions are public, a model may have seen related material. Fresh internal tasks reduce that risk, although they make cross-company comparison harder.

Verifier quality is equally important. A test that accepts one hard-coded string can reward a brittle patch. A test suite that depends on an unreliable external service can fail correct work. Run the reference solution and deliberately broken solutions against every verifier.

Watch for infrastructure bias. Different models may use different command patterns, which can interact with rate limits, package caches, or shell defaults. Track setup failures separately from task failures.

Finally, avoid choosing tasks after seeing which model wins. Freeze the dataset and evaluation plan first. Otherwise ordinary benchmark iteration turns into an expensive way to confirm a preferred answer.

Turn Results Into an Engineering Decision

Rankings are less useful than failure clusters. Group unsuccessful trials by cause: misunderstood requirement, incorrect edit, missing verification, destructive command, timeout, dependency failure, or unsafe behavior.

Then connect the cluster to a product change. A model that solves tasks but skips tests may need a stronger harness contract. Frequent dependency failures may justify a prepared snapshot. Unsafe commands may require tool restrictions and approval gates rather than another prompt paragraph.

Evaluate cost and latency beside quality. A five-point success improvement may be worthwhile for release remediation and wasteful for routine formatting. Route tasks based on consequences instead of declaring one model the permanent winner.

Publish an internal evaluation card with the dataset scope, known gaps, environment, date, and intended decision. State what the benchmark does not measure. This prevents a result collected for terminal maintenance tasks from being reused later as evidence for customer-support quality or autonomous production access.

Revisit the dataset after incidents and difficult human reviews. Add a sanitized task when it represents a failure mode the current suite missed. Retire tasks that no longer match the stack, but keep versioned results so apparent improvement is not caused by quietly removing the difficult work.

The strongest evaluation program ends with a small number of repeatable decisions: which agent handles which task class, what authority it receives, when a human reviews the result, and which evidence must accompany a claimed success.