How Much Does the Agent Harness Matter?
You have picked a model. Now you need to choose the agent that will run it. Most benchmarks cannot help with that decision because they compare models while keeping the agent harness fixed.
We did the opposite. We kept the model slug and tasks fixed, then changed the harness: the prompts, tools, context handling, and execution loop around the model.
Every configuration used deepseek/deepseek-v4-flash-0731, the same ten Terminal-Bench 2.1 tasks, the same Daytona sandbox, and one attempt per task. We compared Ante, Ante-short (Ante with --short-prompt, introduced in our previous post), Pi 0.73.1, OpenCode 1.18.4, and Hermes v2026.7.20.
Full disclosure: Ante is our agent. We chose the tasks before running any harness. All five used high reasoning effort, with other settings left at each harness's defaults.
Here are the three questions this test can help answer.
1. Does the harness change the result?
In this test, yes. Ante passed all ten tasks, Ante-short passed nine, and Pi, OpenCode, and Hermes passed seven each. Errors, timeouts, and failed checks all counted as failures.
Ten tasks are not enough to establish a stable ranking. They are enough to show that the harness can change the result and that a larger test is worth running.
2. Which one should I actually run?
It depends on what matters to you. Ante had the highest pass rate. Ante-short gave up one pass but was faster, cheaper per pass, and used fewer tool calls.
The two Ante configurations ranked first or second on every metric in this chart. Its "overall" score counts pass rate twice and each other metric once. That weighting is only an example; a different priority can produce a different winner.
The practical choice is simple: use Ante when success rate matters most. Use Ante-short when speed and cost matter more. In this test, Ante-short was 16% faster, 18% cheaper per pass, and used 19% fewer tool calls, but passed one fewer task.
| Harness | Passed | Cost / pass † | Agent time / task | Tool calls / task | Binary |
|---|---|---|---|---|---|
| Ante | 10/10 | $0.0359 | 438s | 30.8 | 34.1 MiB |
| Ante-short | 9/10 | $0.0295 | 370s | 25.1 | 34.1 MiB |
| Pi | 7/10 | $0.0372 | 507s | 32.2 | 142.6 MiB |
| OpenCode | 7/10 | $0.0403 * | 512s | 31.8 | 341.2 MiB |
| Hermes | 7/10 | $0.0749 | 694s | 36.2 | 618.3 MiB |
† We applied the same token prices to every harness. This is not the provider's bill. * OpenCode's cost is a lower bound; see below.
3. What does the harness cost to run?
The table uses the same token prices for every harness so the comparison is fair. Actual provider bills can differ. Pi reported a bill of $2.7069, while the same tokens cost $0.2603 with our reference prices. The difference came from provider pricing, not from Pi using 10× more tokens.
With the shared price table, Hermes cost 2.1× as much as Ante per passed task and passed three fewer tasks.
OpenCode's $0.0403 per pass is a lower bound because Harbor 0.20.0 left 186k reasoning tokens out of its total.
Cost is not only about tokens. The harness also changed runtime, memory use, and binary size.
We measured the full sandbox, including the builds and commands each agent launched. Hermes took 694 seconds per task, compared with 370 seconds for Ante-short. Average memory ranged from 118.6 MiB for Ante-short to 847.7 MiB for Hermes, a 7× difference. Binary size ranged from 34.1 MiB for Ante to 618.3 MiB for Hermes, an 18× difference.
What can we conclude?
This is an early result, not a final ranking. We tested ten tasks once with one DeepSeek release, and Ante is our product. One changed result moves the pass rate by ten percentage points, so a larger test with repeated attempts is needed.
The useful conclusion is narrower: keeping the model slug fixed did not keep the result fixed. The harness changed pass rate, cost, speed, tool calls, binary size, and memory use.
When you report a model score, name the harness too.