Skip to content

Reducing Token Usage of State-in-Context Agents using Minification

🕒 Published (v1): 2026-05-31 16:24 UTC · Source: Arxiv · link

Why this paper was selected

Minification reduces state-in-context agent token cost; independent SWE-bench replication

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper replicates the DirectSolve state-in-context agent and applies Python code minification (whitespace removal, comment/docstring stripping, dedentation, identifier shortening, import merging) to the repair context before each LLM call. Minification cuts average input tokens by 42% at a cost of 12 percentage points in SWE-bench Verified resolution rate (50.0% → 38.0%). The work frames lightweight, semantics-preserving source transformations as a practical cost-reduction lever for large-context SE agents.

Problem

State-in-context SE agents (e.g., DirectSolve) aggregate entire repository files into a single repair prompt, where source code accounts for >91% of all tokens consumed. LLM cost scales with context length, and long contexts correlate with degraded reasoning; yet prior work treats context management as an implementation detail rather than a primary research target. No systematic study had evaluated code minification as a first-class context-reduction mechanism in an end-to-end, repository-level agent.

Method

The authors independently re-implement DirectSolve (no public artifact existed): (1) rank files by relevance using file paths alone, (2) aggregate top-ranked files up to a fixed 100,000-token budget, (3) prompt the LLM to produce a SEARCH/REPLACE patch. Minification transforms are applied to retrieved source files before repair:

  • Whitespace removal: strip blank lines and spaces around operators.
  • Comment/docstring removal: delete # comments and triple-quoted docstrings.
  • Dedent: reduce indentation from 4 to 2 spaces consistently; a system instruction forces the LLM to output patches with 4-space indentation to preserve patch applicability.
  • Short identifiers (non-semantics-preserving): rename variables/functions/classes to short aliases with no mapping.
  • Short identifiers (lookup-based): same renaming but prepend a mapping table to the prompt; LLM is instructed to use original names in output; only identifiers whose replacement saves more tokens than the mapping entry costs are renamed.
  • Import merge/remove: consolidate or delete all import statements.

After patch generation, identifier mappings are reversed in SEARCH/REPLACE blocks. A whitespace-tolerant patch matching procedure aligns minified search blocks against original files. Full-benchmark evaluation uses GPT-5-mini; ablations use GPT-4.1. Token counts use OpenAI's tiktoken with GPT-4 tokenizer for cross-run consistency.

Key Contributions

  • Clean-room re-implementation of DirectSolve, achieving 50.0% pass@1 on SWE-bench Verified (vs. originally reported 50.8% with Gemini-2.5-Pro), confirming the method's reproducibility.
  • Token-allocation analysis showing repair-stage source code consumes ~91.8% of all tokens; NL instructions and ranking together are negligible.
  • Five minification transform families for Python integrated into the repair pipeline with patch-applicability safeguards.
  • Empirical evaluation showing stacked minification yields 42% input-token reduction with 12 pp resolution-rate drop, and that the trade-off is stable across difficulty quartiles (~40% token reduction, 10–17% performance drop).
  • Ablation studies isolating individual transform contributions and characterizing failure modes introduced by minification.

Results

  • Baseline (no minification, GPT-5-mini, full benchmark): 50.0% resolution rate, 90,535 avg. input tokens.
  • Stacked minification (GPT-5-mini, full benchmark): 38.0% resolution rate, 52,776 avg. input tokens — 42% token reduction, −12 pp resolution rate.
  • Input tokens dominate repair cost (~93%); token savings translate to proportional cost savings.
  • By difficulty: token reduction consistent at ~40–43% across all quartiles; performance drop ranges 10–17 pp (no change for >4-hour issues, both at 0%).
  • By repository: token reductions range 30–57%; largest performance drops include psf/requests (75% → 25%) and matplotlib (41% → 21%).
  • Single-transform ablations (GPT-4.1, 100-instance subset): remove-docstrings reduces avg. tokens from 87,325 to 68,202 and achieves lowest per-instance cost ($0.1475); dedent causes the largest performance drop (−8 pp) and is excluded from the stacked run.
  • Reduce-operators (whitespace around operators): tokens drop to 80,132, cost $0.1715, minimal performance impact.

Limitations

  • 12 pp resolution-rate drop is non-trivial; minification is not free and the trade-off may be unacceptable for production use cases requiring high accuracy.
  • Dedent excluded from stacked run due to disproportionate performance harm, indicating that indentation-sensitive patches are a hard constraint the approach does not fully solve.
  • Non-semantics-preserving identifier shortening removes naming cues that aid LLM reasoning; the lookup-based variant adds mapping overhead that partially offsets token savings.
  • GPT-4 tokenizer used as proxy across all models; actual tokenization for GPT-5-mini may differ, making absolute token counts approximate.
  • Single-sample patch generation (no majority voting) diverges from some production-quality agent setups, potentially underestimating the baseline.
  • Python-only: minification transforms are language-specific and require re-engineering for other languages.
  • Repository-level variance is large (e.g., psf/requests drops 50 pp), suggesting minification impact is highly repo-dependent.

Relevance to Harnesses / Meta-Harnesses

This paper directly targets the context-management layer of agentic harnesses: the pre-processing transform pipeline that sits between file retrieval and the LLM call. For harness designers, it establishes that source code is the dominant token sink in single-prompt SE agents and that lightweight, deterministic transforms applied at the harness level (not via LLM summarization) can recover ~40% of context budget at acceptable accuracy cost. The lookup-based identifier mapping and patch-reversal logic exemplify how a meta-harness must coordinate input transformation, prompt augmentation, and output post-processing as a coherent pipeline—exactly the concern of harness/meta-harness architectures. The ablation methodology (isolating and stacking transforms) also serves as a template for evaluating context-management modules within a broader agentic scaffold.