Stop Guessing Which Coding Agent to Use — Benchmark It on Your Own Tasks

Every few weeks a new coding-agent leaderboard makes the rounds, and every time the question I actually care about goes unanswered: not “which model tops a public benchmark” but “which model, wrapped in which CLI, does my kind of work well enough at a price I’m happy to pay.” Those are two different questions, and neither shows up on a generic leaderboard built from someone else’s tasks.

So on 2026-07-16 I stopped guessing and measured it. I cloned an open-source framework, aws-samples/sample-agent-cost-bench [1], pointed it at a task set that looks like my real work, and let it report cost, quality, and latency side by side. This is the story of what I ran, what broke along the way, and the two results that changed how I’d choose.

The setup, and the thing that broke

The install was boring in the good way: pip install -e . on Python 3.13 (3.14 was too new for the pinned test dependencies, a small “watch your interpreter version” beat). Before spending on a full matrix, I ran one task with one model as a smoke test. It passed, which told me that particular task-model-tool path worked end to end before I committed real money to it — each distinct runner and verifier path still needed its own check. That habit paid off twice, as you’ll see.

Then I hit the wall. Five of the framework’s tasks verify their output inside Docker containers, and I don’t run Docker Desktop, so I use Finch, Amazon’s open-source container client (it ties together nerdctl, containerd, BuildKit, and a Lima VM, under Apache-2.0) [2]. The first container image build stalled for ninety minutes with zero progress. Worse, every status check I ran then hung too, because they queued behind the wedged build daemon.

The best-supported diagnosis was intermittent network instability inside the Finch VM: some image pulls succeeded, others stalled, and finch build has no pull timeout, so a transient stall became a permanent hang. The evidence pointing there: pulling a small image directly failed once, then succeeded on retry, while a direct request from the host to the same registry returned in a fraction of a second.

I had a choice: keep grinding on the Docker path, or isolate the flaky dependency and get real results now. I chose the second. Ten of the tasks have no Docker dependency at all, so I ran those and deferred the five Docker-verified ones. That’s an honest limitation, not a hidden gap: I ran 10 tasks in the model comparison and 9 in the tool comparison, and the five container tasks are recoverable later with a VM reset, a per-build timeout, and pre-pulled base images. Isolating the flaky dependency instead of letting it block the whole run is the lesson I’d keep.

One more thing broke, and the smoke test caught it. The first time I compared tools, Claude Code failed with $0 cost and a 0% score: “Could not load credentials from any providers.” Its AWS credential hook wasn’t resolving in the run environment. The fix was to export resolved credentials into the environment before launch. Bedrock model access and the effort setting were fine; it was purely credential resolution. Catching that in a one-task smoke run saved me from eighteen wasted runs.

Two questions, one run

The framework answers two questions that people routinely conflate:

• Which model for which task? Hold the tool fixed (I used the Kiro CLI) and vary the model. This is a quality-per-dollar question.
• Which tool for my team, at my budget? Hold the model fixed and vary the CLI around it. This is an agent-stack question: the CLI’s system prompts, context management, and tool policies all differ, not just the price sheet.

Both report the same three numbers together, cost (in USD and each tool’s native unit), pass rate, and latency, because looking at any one in isolation is how you end up with a fast agent that fails half your tasks, or a cheap one you can’t trust.

Two design choices tie this to your work — they buy relevance, not statistical reliability; repeated trials, a versioned task suite, and contamination checks remain your job. First, the task set is yours: the framework can clone a real GitHub repo, pin it to a commit, and verify the result with your own tests, so you’re measuring work that looks like your work rather than a generic puzzle. Second, verification is pluggable (pytest, a Docker image, a custom scorer, or an LLM-judge rubric), so each task is graded the way that task deserves. And the task set is meant to evolve: as your work changes and models improve, you add and retire tasks, and the benchmark stays a mirror of your reality instead of a frozen leaderboard.

Run 1: which model, inside one tool

Three models inside the Kiro CLI, ten tasks each, thirty runs total. The full run cost $3.34 (83.5 credits) and took about 37 minutes, with 27 of 30 runs passing.

*Latency was measured with two to three runs sharing one laptop, so treat it as directional, not clean.

**Two different cost bases, so it’s worth being precise about which is which. Avg $/run is that model’s total spend divided by its ten runs: Sonnet $0.816, Opus $2.108, Haiku $0.416, summing to the $3.34 (83.5 credits at $0.04) above. $/success is the framework’s cost_per_pass, the mean cost of the runs that passed — not total spend divided by successes. The two coincide for Sonnet because it passed everything; they diverge for Opus and Haiku because their failures cost a different amount than their passes. (An earlier version of this table reported the per-run column from the framework’s avg_cost_usd field, which on a ten-task run divides by nine and so overstated each average by about 11%.)

The surprise is in the middle row. Opus 4.8, the most expensive model here, cost roughly five times Haiku per run and still missed a task, a duplicate-device IoT check it scored 25% on, expensively. Sonnet 4.6 passed everything, at a third of Opus’s per-run cost, and was the fastest of the three on this task set. On my tasks, the priciest model did not buy the best outcome. That is exactly the kind of thing a generic leaderboard hides and a benchmark on your own tasks reveals.

Haiku is the honest counterweight: cheapest by far, but it dropped to 80%, missing one task entirely and only half-completing another. Cheap is only cheap if it finishes the job.

Run 2: same model, two tools

Now hold the model fixed (Claude Opus 4.8, running on Amazon Bedrock in both cases) and change only the tool around it: the Kiro CLI versus Claude Code. Nine tasks each, eighteen runs.

*Same caveat: latency was inflated by parallel runs on one machine.

The headline reading, “Claude Code is cheaper and faster,” is true but shallow. Claude Code’s higher average isn’t dragged down by any failure because it passed all nine; Kiro’s average is pulled up by its one miss (the same duplicate-device task Opus struggled with in Run 1). Look at cost per successful task and they’re almost identical: $0.192 versus $0.198. On completed work, these two are neck and neck.

And here’s the finding that changes the verdict: the price you actually pay can flip the winner. The framework converts each tool’s native unit to USD using a rate you supply. For Kiro I used its credits; for Claude Code, the USD it reports directly from Bedrock token pricing. The numbers above use $0.04 per Kiro credit, the published add-on/overage rate. But if you’re inside a paid Kiro plan, the effective in-plan rate is lower: a $20/month Pro plan includes 1,000 credits, which works out to about $0.02 per credit, and every paid tier lands on that same $0.02 (as of 2026-07-16 on kiro.dev/pricing) [3]. At that in-plan rate, Kiro’s credit-based cost roughly halves, and it becomes about half the cost per success of Claude Code rather than even. That comparison is on the cost_per_pass basis in the tables above; if you instead divide each tool’s total spend by its successful tasks, Kiro lands around 20% cheaper rather than half. Two defensible rates, and two defensible cost bases — state which one you mean.

The cross-cutting finding, honestly caveated

One task, geotrack-duplicate-device, failed on Opus 4.8 in both runs (the model comparison and the tool comparison), scoring 25% each time. That consistency across two independent runs makes it look like a genuine Opus weak spot on this particular problem, not a fluke or a tool artifact. Claude Code passed it, though at a higher cost.

Here’s the honest asterisk: that task is graded by an LLM-judge rubric, and the judge is Opus itself. Rubric scores carry judge variance — and, when a model grades its own family’s output, a possible systematic self-grading bias on top — in a way deterministic pytest or Docker checks don’t, so I’d treat the 25% as a strong signal worth investigating, not a hard number. When a result rests on a model grading model output, say so: that’s the difference between a benchmark you can act on and a leaderboard you have to take on faith.

Choosing a plan is a volume decision, not a rate decision

A detail worth pulling out, because it’s a common misread: across Kiro’s paid tiers (Pro, Pro+, Pro Max, Power), the effective per-credit rate is the same ~$0.02. You don’t move up a tier to get a better rate; you move up because you need more monthly credits. Pick the tier that matches your monthly volume, and treat add-on credits ($0.04 each) as the pressure valve when you go over. (Kiro’s GovCloud pricing runs about 20% higher and has no free tier, if that’s your environment.) The benchmark makes this concrete: it forces you to supply your rate, so the USD column reflects the plan you’re actually on rather than a vendor’s headline number.

If you’re running this on AWS

The tool comparison ran Claude Opus 4.8 through Amazon Bedrock in both CLIs. Bedrock is what let me run a frontier Anthropic model on AWS with my own account and credentials, and get per-task USD out the other side. If you want to reproduce this:

• Confirm the model is available in your Region. I verified Opus 4.8 in us-east-1 with a quick aws bedrock list-foundation-models call before starting.
• For Claude Code on Bedrock, resolve your AWS credentials into the environment before launching the CLI (aws configure export-credentials --format env) so its credential hook has something to read. This was the single thing that turned my first 0% run into a 100% one.
• Start with a small task_ids: subset to estimate cost before committing to a full matrix. Benchmarks consume real credits and tokens.

The framework itself is model- and tool-agnostic: it also supports GitHub Copilot, Cursor, and OpenAI Codex as runners. I didn’t run those here (each needs its own subscription and login), so I can’t tell you how they compare, and I’m not going to pretend otherwise.

What I’d tell my past self

Build a small task set that looks like your real work, verify it the way each task deserves, and measure cost, quality, and latency together — then let it evolve. That’s the whole method, and it beats any leaderboard for the only question that matters to you: what should I use, for my work, at my budget?

The framework is on GitHub as aws-samples/sample-agent-cost-bench [1] under MIT-0. Clone it, define three or four tasks that mirror your week, and run it before your next model or tool decision.

What would go in your task set — and would you trust a model to grade the fuzzy tasks, or only the ones a test can check?

Sources

• [1] aws-samples/sample-agent-cost-bench (GitHub, MIT-0) — the benchmark framework used for both runs.
• [2] runfinch/finch (GitHub, Apache-2.0) — the open-source container client behind the Docker war-story.
• [3] Kiro pricing — plan prices, credit allocations, and the $0.04 add-on/overage rate (as of 2026-07-16).

#AI #CodingAgents #Kiro #AmazonBedrock #Benchmarking

About the Author

Stefan Christoph is a Principal Solutions Architect at AWS, focused on agentic AI, media & entertainment, and helping builders move from demo to production. He writes about AI architecture, developer productivity, and the future of software.

This is a personal blog. Opinions expressed here are my own and do not represent the views or positions of my employer.

Learn more →

• AI
• CodingAgents
• Kiro
• AmazonBedrock
• Benchmarking