Skip to content

Unlocking Model Potentials Through Adaptive Multi-Agent Scaffolding for Efficient Issue Resolution

🕒 Published (v1): 2026-06-24 07:48 UTC · Source: Arxiv · link

Why this paper was selected

Adaptive scaffolding for ambiguous issue resolution; directly models harness-level workflow design

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

icat-agent is a decentralized multi-agent scaffolding for automated GitHub issue resolution that replaces shared global context with event-based message passing between isolated Explorer, Patch Editor, and Validator agents. A rubric-based issue quality checker routes well-defined issues directly to parallel patch/test generation while ambiguous ones first pass through an AST-powered Explorer. On SWE-bench Pro, icat-agent + GPT-5.4-xhigh achieves 67.4% resolution, outperforming the prior best (mini-SWE-agent, 59.1%) by 8.3 pp while costing ~47% less per instance than the Claude Code baseline.

Problem

Monolithic single-agent scaffolds (e.g., SWE-agent) suffer from monotonically growing shared context that causes contextual degradation and two forms of context poisoning: (1) test overfitting — the reproduction test is written before the patch and is too narrow, so the patch satisfies the test without fixing the underlying bug; (2) patch overfitting — the patch is written first and the subsequent test is crafted to pass the patch rather than verify correctness. Existing multi-agent designs (orchestrator/sub-agent, team-lead) do not fix this because the central coordinator still accumulates summaries from all sub-agents, propagating bias downstream. Additionally, 35–41% of SWE-bench issues lack explicit buggy-file/function pointers, causing agents to waste budget on unguided exploration under uncertainty.

Method

The system formalizes trajectory length as \(|H| \propto \frac{\text{Complexity}(B)}{F}\), where \(F \in (0,1]\) is issue quality (lower = more ambiguous). Context utility is \(U(\mathcal{A}, t) = \max_k \frac{C_{A_k}(t)}{W}\); icat-agent minimizes this by ensuring each agent receives only a small structured signal \(\sigma\) from peers (\(|\sigma| \ll |H_k|\)), yielding:

\[U(M,t) = \max\!\left(\frac{|I_F|+|H_E|}{W},\; \frac{|\sigma_{E\to V}|+|H_V|}{W},\; \frac{|\sigma_{V\to P}|+|H_P|}{W}\right)\]

Four components:

  1. Issue Quality Checker — evaluates four binary criteria (buggy file named? buggy function named? fix strategy described? reproduction info present?) and routes: all four true → high quality (skip Explorer, start parallel execution); any false → low quality (invoke Explorer first).
  2. Explorer Agent — uses AST-based tools (tree-sitter, 100+ languages) to localize fault to file/class/method, extract call chains, and identify suspicious lines. Produces a structured context summary shared with Validator and Patch Editor; contains no test code or patches.
  3. Validator Agent — generates reproduction tests (expected to fail pre-patch, pass post-patch) and regression tests. Communicates only pass/fail execution results + failure-contributing statements to the Patch Editor, never raw test assertions. May strengthen but not weaken tests.
  4. Patch Editor Agent — generates and revises patches. Receives validation results but never sees test code, preventing patch overfitting. Communicates generated patches back via message bus.

Validator and Patch Editor run in parallel, iterating until the patch passes or the token budget \(B\) is exhausted.

Key Contributions

  • Rubric-based adaptive workflow routing: binary quality score gates parallel vs. exploration-first execution paths, optimizing accuracy-to-cost ratio per issue.
  • Context-isolated multi-agent coordination: no shared global context; structured inter-agent signals replace full trajectory sharing, eliminating context poisoning by construction.
  • Asymmetric information hiding in Validator↔Patch Editor loop: Validator withholds test code from Patch Editor and pass/fail results from Validator's view of patch internals, preventing both overfitting directions simultaneously.
  • AST-powered Explorer generalization: tree-sitter tooling enables cross-language fault localization without language-specific heuristics.
  • Empirical scaffold-dominance evidence: same backbone LLM resolves markedly more issues under icat-agent than under SWE-agent, mini-SWE-agent, or Claude Code.

Results

  • SWE-bench Verified (500 instances): +3.6–8.4% over all baselines across all backbone LLMs tested (MiniMax M2.5, GPT-5-mini, Claude Sonnet 4.5).
  • SWE-bench Pro (731 instances):
  • icat-agent + GPT-5.4-xhigh: 67.4% vs. mini-SWE-agent + GPT-5.4-xhigh: 59.1% (+8.3 pp)
  • icat-agent + Claude Sonnet 4.5: +18.5% over SWE-agent + Claude Sonnet 4.5
  • icat-agent + GPT-5.4-xhigh: +6.3% over Claude Code + GPT-5.4-xhigh
  • Cost: icat-agent averages $1.27 (Sonnet 4.5) and $1.49 (GPT-5.4-xhigh) per instance; Claude Code baseline costs $2.67 — a $1.18 average saving per instance.
  • Difficulty robustness: gains hold across Easy/Medium/Hard tiers; most pronounced on Medium and Hard.
  • Cross-language: largest performance delta in non-Python languages (TypeScript, JavaScript, Go).

Limitations

  • Quality checker uses a conservative all-or-nothing binary rule (all four criteria must be satisfied for high quality); this may over-trigger Explorer for issues that are only marginally underspecified, incurring unnecessary cost.
  • Evaluation confined to SWE-bench Verified and SWE-bench Pro; both are curated, filtered benchmarks — the paper acknowledges in-the-wild issues are likely harder and more ambiguous.
  • Relies on tree-sitter for AST exploration; correctness of fault localization for dynamically typed or heavily metaprogrammed code is untested.
  • Budget \(B\) is a fixed token ceiling; there is no adaptive budget allocation between agents based on mid-run difficulty signals.
  • No ablation published on the contribution of the quality-routing step alone vs. the context-isolation step alone.

Relevance to Harnesses / Meta-Harnesses

icat-agent is a direct instance of a meta-harness: a scaffold that wraps multiple specialist agents (Explorer, Validator, Patch Editor) behind a routing layer that chooses the execution topology at runtime based on input characteristics — precisely the pattern that defines adaptive meta-harnesses. The rubric-based quality checker acting as a dynamic router and the event-driven message bus as the coordination substrate are reusable architectural primitives applicable far beyond SWE tasks. The paper's central empirical claim — that scaffold architecture is "a first-class determinant of agent performance, on par with model capability itself" — directly validates the meta-harness research program. The asymmetric information hiding between sub-agents (Validator withholds test code; Patch Editor withholds patch internals) is a novel isolation technique with broader applicability to any harness where independent verification is needed.