Coding Agent Study Finds 64% Cost on Failed Runs
Corrected Tuesday, 17 November 2026: The share was of the list-price cost of the scored runs, not of all inference cost.
A large-scale evaluation of AI coding agent harnesses found that most inference costs came from runs that did not succeed.
- The evaluation used 83 billion tokens across 27,280 agent runs.
- It tested 22 harness configurations with 5 models and 62 tasks.
- 64.5% of the list-price cost of the scored runs went to trials that did not pass.
- The cost per pass varied by 1.9 to 4.9 times between the cheapest and most expensive harness for a given model.
- The number of steps a harness takes correlates with its cost per pass.
Who it affects People choosing coding harnesses and models for code tasks.
What to do
Test candidate harnesses with your model on your own tasks when changing models.
Source: runta.com ·