Code Isn't Memory: A Structural Codebase Index Inside a Coding Agent¶

🕒 Published (v1): 2026-06-21 10:10 UTC · Source: Arxiv · link

Why this paper was selected

Controlled study of structural indexing inside a fixed harness; actionable retrieval finding

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper runs a controlled, within-harness ablation of a structural codebase index (semantic + lexical + call-graph) inside the SuperCoder coding-agent harness, holding Claude Opus 4.7 fixed across all arms and 91 instances over three seeds. Adding the index moves file-level localization (View B acc@5) from 44.3% to 84.5% and resolve from 41.9% to 50.4%, with no per-cell cost penalty and lower cost per solve than the agentic-grep comparator. The deployment question shifts from "is the index too expensive?" to "does the workload include multi-file changes where structural ranking pays off?"

Problem¶

Most coding-agent evaluations either compare whole harnesses—where retrieval is confounded with prompt design, tool surface, and control loop—or benchmark retrieval components in isolation against acc@k without downstream agentic resolution. No prior work reports a leak-audited, model-controlled causal ablation of a structural codebase index inside a single open-source harness; consequently, the cost-vs-resolve trade-off of such an index has not been cleanly quantified.

Method¶

Three-arm experiment on 91 instances (34 Go, 20 Java, 37 Python) drawn from SWE-PolyBench Verified and SWE-bench Pro, run with three seeds each:

SC-ON: SuperCoder harness with codebase_search and codebase_graph tools enabled.
SC-OFF: Identical harness, identical prompts; only those two tools removed from the schema.
OpenCode: Independent open-source harness using ripgrep + file reads; no structural index; same model, same 30-minute wall-clock cap.

The context engine (SC-ON only) builds a per-repository index once via tree-sitter AST parsing, then updates incrementally via Merkle-tree diffs. It combines three indices: a BM25 lexical index over identifiers/tokens, a call-graph index of definition/caller/callee edges, and a vector index of code-chunk embeddings. At query time, codebase_search issues hybrid retrieval (fuse + rerank across all three) and codebase_graph traverses the call-graph index by symbol.

Integrity protocol: fail-closed per-cell sandbox with git gc --prune=now to physically scrub gold-fix objects before agent execution, plus a post-run S1 audit of all 386 traces for residual git_history_leak events. A dual-view localization extractor defines View B (agent-targeted): paths returned in engine result lists do not count as surfaced unless the agent subsequently grep'd, read, or edited them—eliminating the asymmetry that inflates surfaced sets for arms with engine-list output.

Statistical tests use the Wilcoxon signed-rank test on per-instance mean-of-seeds pass@1, two-sided; paired denominators are 80 (SC-ON vs. SC-OFF) and 78 (SC-ON vs. OpenCode).

Key Contributions¶

First leak-audited, model-controlled, within-harness causal ablation of a structural codebase index in a coding agent.
View B localization metric: strips engine-result-list provenance to create a symmetric, agent-trajectory-grounded acc@k applicable across harnesses with and without indices.
Reframes the deployment question: index cost is not the bottleneck (lower $/solved than agentic grep); workload fit (multi-file structural changes) is the relevant criterion.
Released artifacts: per-cell exclusion ledger, leak-audit script (scoring/leak_audit.py), localization extractor (scoring/localization.py), and full results database.

Results¶

Within-harness ablation (SC-ON vs. SC-OFF, n = 80 pairwise): - View B acc@5: 44.3% → 84.5% (Wilcoxon $p < 0.0001$) - Resolve: 41.9% → 50.4% (Wilcoxon $p = 0.003$) - Per-cell cost: statistically null difference - $/solved: SC-ON lower than SC-OFF

Cross-harness validity check (SC-ON vs. OpenCode, n = 78 pairwise): - Resolve: 50.4% (SC-ON) vs. 45.3% (OpenCode); $p = 0.087$ (not significant) - View B acc@5: 84.5% vs. 75.3%; $p = 0.080$ (not significant) - Mean $/cell: SC-ON $2.30 vs. OpenCode $2.92; SC-ON $/solved favorable

Integrity: Post-exclusion metric shifts are <1 pp on all outcomes; no metric ordering flips.

Limitations¶

Single model (Claude Opus 4.7) throughout; results may not generalize to other backbones.
Language coverage is Go, Java, Python only; JavaScript and TypeScript excluded.
Effective denominators (75–80 instances) leave resolve-level paired tests underpowered.
All 5 provider_truncation exclusions fall on SuperCoder arms (none on OpenCode), introducing a potential asymmetric exclusion bias.
Ecological validity threat from formal GitHub issue text vs. realistic user phrasing (Garg et al.) is acknowledged but not addressed by the hardening protocol.
The context engine's backend service is internal and not released; the ablation surface covers tool availability, not the index implementation itself.
Closed-source harnesses (Claude Code, Cursor, Windsurf) are out of scope by design.

Relevance to Harnesses / Meta-Harnesses¶

This paper is a direct empirical study of harness internals: it isolates a single component (a retrieval subsystem) by toggling it within an otherwise identical harness, producing a causal estimate of that component's contribution to end-to-end resolve and cost. The experimental design—fixed model, fixed everything-else, single flag change—is a template for within-harness ablation methodology applicable to any pluggable harness component (context windows, tool surfaces, compaction strategies). The View B localization metric addresses a cross-harness measurement asymmetry that would otherwise make index-equipped and grep-only harnesses incommensurable, which is a prerequisite for any meta-level comparison across harness configurations. The integrity protocol (fail-closed sandbox, leak audit, public exclusion ledger) sets a reproducibility standard directly relevant to anyone building or evaluating harnesses on SWE-bench-family tasks.