Hacker News·5 min read·hard

In 27,280 coding agent runs, 64% of inference spend went to runs that failed

E
Edward40
In 27,280 coding agent runs, 64% of inference spend went to runs that failed
✦AI Summary

A large-scale analysis of 27,280 coding agent runs reveals that 64% of inference spending is wasted on failed attempts. The study suggests that developers must carefully match specific harnesses to models to optimize cost-efficiency and performance.

Why it matters

As AI agent usage grows, the high cost of failure and the complexity of model-harness compatibility represent significant financial and operational hurdles for developers.

✦Dive DeeperCreate a free account to unlock

We partnered with Baseten to build the largest-scale harness leaderboard using 83B tokens across 27,280 agent runs. We wanted to answer two questions: when you switch your model, should you consider switching the harness too? What is the best harness for each model?

Previously, our v1.0 FrontierHarness leaderboard ranked harnesses on a single model, Kimi K3.

V1.1 expands the benchmark to 22 harnesses × 5 models × 62 tasks × 4 attempts, 76× the size of v1.0.

Through our experiments, we found that you should consider switching harnesses when switching models to maximize your experience.

Each model had a different best-value (pass rate vs. cost) harness: Codex CLI with GLM-5.3, Pi with GLM-5.3 Flash, ZCode with DeepSeek V4.1 Flash 1017, Oh My Pi with Muse and Alma with Kimi-K3. Cost ranks moved far between models.

Continue reading on Headlinne

Create a free account to read the full article.

Read full article →
technologybusinessai
✦

Get smarter about the news

Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.

Create free account

Already have an account? Sign in