Skip to content

The Hidden Footprint: Making Storage a First-Class Metric for LLM Agent Evaluation

🕒 Published (v1): 2026-07-13 06:44 UTC · Source: Arxiv · link

Why this paper was selected

Introduces storage footprint as agent eval metric; novel axis overlooked by all current benchmarks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Current LLM agent benchmarks track task accuracy, reliability, and inference cost but completely ignore the persistent storage each run leaves on disk. AgentFootprint is the first cross-framework benchmark measuring post-run storage footprint across eight major frameworks, revealing up to 15.7× disparity in retained bytes among equally accurate systems. A content-addressed compactor demonstrates that 4.8–32.7× storage reduction is achievable without sacrificing conversation reconstructability.

Problem

No existing agent benchmark or evaluation framework measures the persistent storage footprint—logs, checkpoints, context snapshots, debug traces—that an agent run leaves on disk. This omission matters for edge deployment, data-retention compliance, and fleet-scale accumulation (e.g., a 10⁴-round/day fleet could accumulate ~560 GB daily). A secondary methodological problem is that naïve byte-level measurement dramatically understates duplication: serialization artifacts (SQLite 4 KB page fragmentation, JSON escape sequences) cause raw chunk-matching to report \(D = 1.01\) on a store that logically holds the same content 12× over.

Method

The authors define six serialization-aware metrics per (framework, task) run: - \(S_\text{total}\): retained bytes (workspace delta + framework residue) - Composition: channel breakdown (checkpoints, event logs, conversation stores, other) - \(D\): duplication factor \(S_\text{total}^\text{logical} / S_\text{unique}\), computed over logical content streams (SQLite cells, JSONL records) rather than raw files, with content probes matched under raw, JSON-escaped, and double-escaped encodings - \(\alpha\): growth exponent from fitting \(\log S_\text{total} \sim \alpha \log T\) over a repeated-observation stress task at \(T \in \{25, 50, 100, 200\}\) rounds - \(C\): compressibility ratio under zstd -19 with 128 MB long-range window (syntactic "free lunch" baseline) - \(R \in \{0,1,2,3\}\): conversation-history reconstructability score, validated by field-by-field reconstruction against a logging proxy (194/194 calls exactly reconstructed for all \(R=3\) frameworks)

Eight frameworks (LangGraph, AutoGen, CrewAI, SmolAgents, OpenAI Agents SDK, LlamaIndex, Agno, InfiAgent) are evaluated across 1,061 sandboxed runs with per-run filesystem isolation (snapshot before, diff after). A fixed-trace control replays the same trajectory through each framework's adapter to isolate persistence-layer amplification from behavioral differences. A content-addressed store (CAS compactor) using FastCDC chunking + SHA-256 serves as a headroom demonstration.

Key Contributions

  • Six-metric suite (Stotal, composition, D, α, C, R) with a methodology showing serialization masks duplication by up to 12× in naïve measurement
  • AgentFootprint benchmark: 509-run main study + 552 replication/variant runs across eight frameworks with container-validated sandbox isolation
  • Finding that frameworks at identical 100% task accuracy differ by 15.7× in retained bytes (LangGraph 5.10 MB vs. CrewAI 0.32 MB per task on a 600 KB corpus)
  • Growth regime characterization: full-history frameworks exhibit \(\alpha\) up to 1.95 (200 rounds over a 2 KB file → 323 MB); windowed/bounded designs stay near-linear (\(\alpha \approx 0.98\)–\(1.20\))
  • In-the-wild study of 108 instance-normalized SWE-bench Verified submissions: per-instance volume spans 1,617× with no detectable correlation with resolve rate (\(\tau_b = 0.080\), \(p = 0.222\))
  • CAS headroom proof: 4.8–32.7× size reduction with all reconstructability scores and restore checks preserved

Results

  • Equal-accuracy dispersion: among five 100%-accurate persisting frameworks, defaults span 15.7× (bootstrap 95% CI [15.0, 16.3]); LangGraph 5.10±0.58 MB vs. CrewAI 0.32 MB per task
  • Duplication masked by serialization: LangGraph's checkpoint store reports \(D = 1.01\) under raw chunking but \(D = 12.1\) under logical-stream extraction; zstd compresses it 142× (C=136), confirming the duplication was present
  • Superlinear growth: LlamaIndex (\(\alpha=1.88\pm0.10\)), AutoGen (\(\alpha=1.95\pm0.18\)), LangGraph (\(\alpha=1.74\pm0.01\)) all fit \(\alpha\ge1.72\); LangGraph accumulates 323±26 MB at T=200; windowed InfiAgent reaches only 1.4–36.4 MB at the same horizon
  • Residue dominates: on write tasks, framework residue exceeds delivered output files by up to 24,033×
  • Fixed-trace isolation: same trajectory replayed through each framework yields sizes differing by 6.7×, attributing amplification to the persistence layer
  • CAS reduction: AutoGen 2.49 MB → 0.07 MB (32.7×); LangGraph 5.10 MB → 0.50 MB (10.3×); all R=3 scores preserved
  • SWE-bench wild study: 108 submissions span three orders of magnitude in per-instance volume; no correlation with resolve rate

Limitations

  • Growth exponents are fitted over only four horizons (T=25–100–200) with three seeds; the paper explicitly states this is insufficient to distinguish candidate functional forms—\(\alpha\) describes superlinear growth over the measured range, not a scaling law
  • Fairness protocol evaluates documented default configurations, not capability-normalized efficiency; frameworks with higher R scores (more recovery capability) retain more by design, so equal-R comparisons are needed for fair comparison but are only partially provided
  • Fixed-trace control and CAS demonstration are existence proofs, not proposed production systems; no runtime overhead of CAS is reported
  • The metric suite covers post-run local retention; distributed or cloud-side persistence (e.g., LangSmith, Langfuse) is out of scope
  • Doubly-nested JSON encodings are only partially detected by the content-probe methodology
  • InfiAgent has no persistence-disable switch, requiring a post-hoc channel-exclusion ablation rather than a clean ablation

Relevance to Agentic AI / LLM Agents

As agentic systems move from prototype to production, storage footprint joins inference cost and accuracy as a deployment-gating resource dimension—yet it has been entirely absent from the evaluation literature this work surveys. The 15.7× disparity among equally accurate frameworks and the superlinear growth regimes (\(\alpha\) up to 1.95) directly impact whether an agent can run on-device, satisfy data-retention regulations, or scale to fleet workloads. For researchers building or benchmarking agent frameworks, the serialization-aware metric suite and the CAS headroom result establish a concrete, reproducible methodology for reporting storage alongside the currently standard axes (accuracy, cost, reliability). The SWE-bench trajectory analysis further shows that current agent submissions already exhibit three-orders-of-magnitude volume spread uncorrelated with capability, making this a tractable and impactful axis for future leaderboard standardization.