Skip to content

ReCUBE: Evaluating Repository-Level Context Utilization in Code Generation

🕒 Published (v1): 2026-03-26 08:04 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ReCUBE is a benchmark that evaluates LLMs on their ability to reconstruct a masked Python file using only the remaining repository files, dependency specs, and documentation as context. The authors also introduce the Caller-Centric Exploration (CCE) toolkit — dependency-graph-based agentic tools that steer agents toward caller files before reconstruction. Even GPT-5 achieves only 37.57% strict pass rate, and CCE-augmented agents consistently outperform all baselines.

Problem

No existing benchmark isolates repository-level context utilization from other coding skills: prior work tests GitHub issue resolution, next-line completion, or QA over repos, all driven by natural-language prompts. There is no prompt-free, contamination-free, manually curated framework that forces models to leverage the repository structure itself as the sole signal for code generation.

Method

Benchmark construction. Twenty GitHub repos (>10K stars, >1K forks, created after January 2025) are manually decomposed into 40 functional subsets. For each subset, up to 10 central Python files are selected as targets (\(N=366\) instances). Each target file \(f_i\) is masked by removing imports and replacing function bodies with raise NotImplementedError while preserving signatures and docstrings. The full context \(C_{f_i}\) = masked codebase + PyPI dependency specs + Markdown docs, wrapped in XML-style delimiters.

Test case generation. Usage-aware unit tests are generated with Claude Opus 4.1 by analyzing in-file and cross-file call patterns, then validated against gold implementations in a Docker testbed (10,785 tests total; 58.7% external cross-file, 41.3% internal).

Metrics. $\(\text{SPR} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}[P_i = T_i] \times 100\)$ $\(\text{APR} = \frac{1}{N}\sum_{i=1}^N \frac{P_i}{T_i} \times 100\)$

Caller-Centric Exploration (CCE) toolkit. A heterogeneous directed graph indexes each instance with four entity types (directory, file, class, function) and four edge types (contain, import, invoke, inherit). Three tools expose this graph to agents: (1) Caller Patterns — which files invoke the target and call frequencies; (2) Inheritance — class hierarchy; (3) Similar Files — BM25 identifier overlap + filename subtoken overlap + AST structural similarity. A fourth tool allows inline Python scripting against the graph; a fifth does syntax validation.

Experimental settings. Four settings evaluated: Full-Context, Full-Context + CoT, bash-only Agent (Mini SWE-agent), and Agent + CCE toolkit, across eight models.

Key Contributions

  • ReCUBE benchmark: 366 prompt-free, manually curated, contamination-free instances; up to 111K token context.
  • ReCUBE-LARGE: 138-instance extension merging full repos, up to 338K tokens.
  • Usage-aware test case generation protocol (internal vs. external split).
  • Empirical finding that caller coverage correlates with pass rate on hard tasks (Spearman \(\rho = +0.408\), \(p=0.035\)).
  • CCE toolkit: dependency-graph tools integrated into agentic loop to surface caller context before reconstruction.

Results

  • GPT-5 (full-context): 37.57% SPR / 60.43% APR — best in full-context setting.
  • Internal test cases consistently ~9.55% SPR higher than external across all models.
  • CoT degrades GPT-5 (−0.28% SPR), Devstral (−2.54%), Qwen3-Coder (−0.48%); helps GPT-OSS 20B (+1.52%) and GPT-5 Mini (+0.55%).
  • Bash-only agent hurts models with large context windows: GPT-5 Mini loses −2.21%/−3.46% SPR/APR vs. full-context.
  • Agent + CCE (GPT-5 Mini): 38.78% SPR — surpasses GPT-5 full-context (+5.35% over GPT-5 Mini full-context, +1.21% absolute over GPT-5).
  • CCE improvements are concentrated on hard/hardest difficulty quartiles; easy tasks see marginal or negative returns.
  • Devstral + CCE: +4.00% SPR / +8.25% APR over full-context baseline.

Limitations

  • Only Python repositories; cross-language generalization untested.
  • Repository selection criteria (stars/forks) may skew toward popular AI-adjacent tooling (LLM services, MCP/tool frameworks dominate the domain distribution).
  • CCE graph is static per instance; it does not update as the agent writes code.
  • Test cases are LLM-generated (Claude Opus 4.1), introducing potential quality ceiling.
  • ReCUBE-LARGE results are not fully reported in the provided text; analysis of behavior at 338K context is limited.
  • CoT findings are model-specific and may not generalize as instruction-tuning choices vary.

Relevance to Harnesses / Meta-Harnesses

ReCUBE is directly relevant as a benchmark harness for evaluating agentic code-generation pipelines: it specifies containerized Docker/Apptainer environments, structured prompt formats, and an automated test execution harness that scores reconstructed files against gold implementations. The CCE toolkit exemplifies a meta-harness pattern — a layer of tooling (graph indexing, caller-surface queries, syntax validation) scaffolded around a base agent to orchestrate its information-gathering loop, precisely the design pattern that distinguishes meta-harnesses from single-agent systems. The finding that vanilla agentic exploration degrades with large context while a structured exploration harness (CCE) recovers and exceeds performance is strong empirical support for the value of harness-level orchestration over raw LLM capability scaling.