Skip to content

Token Reduction Is Not Cost Reduction

🕒 Published (v1): 2026-07-13 21:10 UTC · Source: Arxiv · link

Why this paper was selected

Shows token reduction doesn't equal API cost reduction; counterintuitive result every agent-builder must know

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Token-count reduction from context-compression layers in API-based coding agents does not reliably translate to lower provider-billed cost. In a pre-specified paired campaign of ~2,900 Claude Code runs, a 38% reduction in raw tool-output tokens coexisted with a +6.8% cost increase. The paper proposes "success-adjusted billed cost" as the correct evaluation metric.

Problem

Context-reduction systems for coding agents (compressors, retrieval rankers, API proxies) are almost universally evaluated by tokens removed, yet three mechanisms break the naive equivalence to cost savings: (1) prompt-cache pricing discounts cached tokens far below nominal input rates, so removing cached text saves less than expected; (2) agents that lose needed context re-issue tool calls, retransmitting the full growing prefix; (3) compression can corrupt non-model consumers (e.g., shell pipelines, patch-application anchors), causing task failures that raise cost-per-success.

Method

The authors run a pre-specified, hash-frozen, block-randomized, paired campaign ("authoritative campaign") across: - 2,908 runs (2,848 analyzed), 103 tasks, 7 repositories (6 pinned OSS + 1 synthetic monorepo) - 3 models: Haiku 4.5, Sonnet 5, Opus 4.8; up to 5 effort levels - 4 arms: Baseline Claude Code, RTK (deterministic hook compressor), RTK-ML (RTK + 9 flag-gated ML capabilities including BGE embedding re-ranking and AST-based structural retrieval), Headroom v0.27.0 (API-boundary proxy)

Cost is taken from provider-returned total_cost_usd (not estimated), cross-checked against usage fields and Anthropic's pricing schedule. A four-component cost model is: $\(\hat{C} = \frac{1}{10^6}\bigl(T_\text{unc}\,p_\text{in} + T_\text{cw}\,\mu_w\,p_\text{in} + T_\text{cr}\,\mu_r\,p_\text{in} + T_\text{out}\,p_\text{out}\bigr)\)$ with \(\mu_w = 1.25\), \(\mu_r = 0.1\) (five-minute cache TTL). The primary outcome is cost per successful execution (CPS). A supplementary single-shot grounded-completion study on 40 SWE-bench-derived Go tasks (4 arms × ~29–40 rows) measures patch-apply and test-resolve rates under compression.

Key Contributions

  • Cost anatomy: cache creation + cache reads account for ~87% of reconstructed four-component cost (~80% of actual bill); generated output is a small minority; an explicit 8.7% dollar-weighted residual is unattributable from retained telemetry.
  • Decoupling evidence: RTK's 38.4% reduction in estimated raw tool-output tokens coexists with a paired billed-cost increase of +6.8% (95% CI [+2.8%, +11.3%]); per-task Pearson \(r = 0.15\) (CI crossing zero) between tool-output reduction and billed-cost change.
  • Two documented failure modes: (i) ranked-search output piped into shell counting pipelines expecting raw grep lines silently produces wrong answers; (ii) compression destroys verbatim edit anchors that patch application requires, reducing SWE-bench Go patch application from 27/40 to 15/40.
  • Marginal token cost model: a token delivered at turn \(t\) costs approximately \((1.25 + 0.1 \times N_\text{future reads,t})\,p_\text{in}\); an early-turn token is more addressable than a late one.
  • Layered evidence taxonomy (L1–L8 from compression ratio to trajectory/failure analysis) and benchmark-design recommendations for trustworthy efficiency claims.
  • Trajectory-level accounting of 13,620 deterministically classified assistant turns showing where savings are created and repaid.

Results

  • Cache dominance: cache creation + reads = ~87% of reconstructed cost (~80% of actual bill) across all 2,848 runs.
  • RTK vs. Baseline: +6.8% paired cost (95% CI [+2.8%, +11.3%]) despite 38.4% raw tool-output token reduction; per-task \(r = 0.15\) (CI crossing zero).
  • Headroom: billed effects and trajectories measured but not causally attributed (black-box proxy).
  • SWE-bench grounding study (single-shot, \(n \approx 29\)–40):
  • Byte-exact grounded raw arm: 5/29 solves (up from 1/29 context-only baseline).
  • Grounded compressed arm: descriptively fewer solves than grounded raw; 3 tasks solved by grounded raw were not solved by grounded compressed (0 reverse); exact paired \(p = 0.25\).
  • Compression reduced patch application from 27/40 to 15/40 (anchor-destruction failure).
  • Haiku 4.5: unattributed residual scales with thinking-effort setting.
  • Reconstruction accuracy: median per-run residual ≈1% using published Anthropic prices.

Limitations

  • Workloads are synthetic/seeded repositories plus 6 OSS codebases; generalization to diverse production workloads is unvalidated.
  • Raw vs. delivered tool-output token counts are estimates from a local BPE tokenizer (tiktoken o200k_base), not the provider tokenizer.
  • Per-token addressability (turn of entry × future cache reuse) is not measured; Eq. (4) is presented as a bounded conceptual model.
  • Headroom is a black box: internal mechanisms cannot be attributed.
  • Grounding study is single-shot with small \(n\) (29–40 per arm); paired \(p\)-values are not significant.
  • 8.7% dollar-weighted cost residual is unattributed; on Haiku 4.5 it scales with effort but the mechanism is unknown.
  • The broader exploratory spend (~5,493 runs, ~$643 total) is excluded from all statistical claims but represents a large search process that may introduce selection pressure on which campaigns were retained.

Relevance to Agentic AI / LLM Agents

This paper directly challenges a foundational assumption in agentic system optimization: that reducing tokens in tool outputs is a reliable proxy for reducing operational cost. For researchers building or evaluating LLM agents with tool-use loops, the cache-dominance finding (\(\approx\)87% of cost in cache traffic) reframes the optimization target entirely—compressing already-cached context saves far less than a raw token count implies. The closed-loop failure mode (compression → missing context → extra turns → full prefix re-transmitted) is a fundamental architectural concern for any agentic system with iterative tool calls, and the anchor-destruction failure mode has direct implications for code-editing agents that rely on verbatim file fragments for patch application. The proposed "success-adjusted billed cost" metric and layered evidence standard (L1–L8) offer a reusable evaluation framework for the field.