Measuring the Wrong Number: Cost-Aware, Correctness-Gated Evaluation of Efficiency Claims in Multi-turn Agentic Coding

Abstract

A fast-growing ecosystem of plugins, skills, and prompt packs promises to make AI coding agents cheaper. The most popular single artifact—a "talk-like-a-caveman" skill for a widely used coding agent—has ~92,000 GitHub stars (as of 2026-07) and advertises a 65% token reduction. We show that such claims, though not fabricated, measure the wrong number. Efficiency tools are validated on single-shot question answering and report output-token reduction, whereas a real agentic coding session is dominated by input and cached context re-read every turn; the model's own output is only ~0.6% of session tokens in aggregate and ~20% of the dollar cost. We introduce a reproducible, cost-aware, correctness-gated A/B harness that installs each tool faithfully, runs repeated trials over multi-turn coding tasks, breaks the bill down by token class, and gates every trial on a structural correctness check. Across 140 runs on two widely used efficiency tools — the two most-starred in this category — neither the token claim nor the money behind it survives. Output falls far short of the advertised 63–75%, which an industry test had already found; the finding we add is that the dollar bill does not move at all, coming out statistically flat with confidence intervals that exclude any saving close to the advertised figures. The ceiling is also structural: output is only about a fifth of the bill, so even a perfect 65% output cut caps bill savings near ~13%. We argue that trustworthy evaluation of agent tooling must measure cost on realistic multi-turn sessions with a correctness gate, and we release the recorded measurements with a self-checking script that reproduces every number.