Agent4cs: A Multi-agent System for Code Summarization in Large Hierarchical Codebases

🕒 Published (v1): 2026-07-01 19:41 UTC · Source: Arxiv · link

Why this paper was selected

Multi-agent hierarchical code summarization; scales agent teams to large structured codebases

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Agent4cs is a three-agent harness for bottom-up hierarchical code summarization in large, obfuscation-heavy repositories that exceed LLM context limits. It coordinates a summarization agent, a keyword-extraction agent, and a quality-assurance (QA) agent across folder layers, progressively aggregating summaries from function level up to repository root. Evaluated on seven frontier LLMs across six real-world repositories, it achieves +8% semantic similarity and up to +38% normalized keyword coverage over structured prompting baselines.

Problem

Single-model and flat-prompting approaches to code summarization fail to exploit the hierarchical folder structure of industrial repositories, which routinely exceed 300K tokens and use obfuscation (identifier renaming, control-flow alteration). Existing datasets provide ground-truth only at function level; no open-source baseline handles mid-level folder layers or produces persistent, systematic repository-wide documentation.

Method

Agent4cs implements a bottom-up, three-layer agent pipeline:

1. Function level: A summarization agent generates a draft summary for each code file; a QA agent critiques it and returns feedback; the summarization agent revises. Optionally, AST structure is injected into the prompt.
2. Folder level (grandparent-parent-child framework): Starting from the deepest summarized layer, a keyword-extraction agent runs TF-IDF (scikit-learn) over child-folder summaries to produce a condensed keyword set. This set plus the parent-folder summary are fed as context to the summarization agent to produce a draft grandparent-folder summary. The QA agent then reviews the draft; the summarization agent iteratively refines it. This three-folder-layer sliding window repeats upward until the repository root is reached.

Keyword compression is the explicit mechanism for handling context-window constraints: rather than concatenating all child summaries (infeasible at depth \(\geq 7\)), only extracted keywords propagate upward.

Key Contributions

• First multi-agent framework targeting hierarchical, repository-level code summarization.
• Bottom-up aggregation strategy that explicitly propagates cross-folder context via TF-IDF keyword extraction rather than raw summary concatenation.
• Evaluation framework for reference-free hierarchical summarization: semantic similarity (SentenceBERT cosine), keyword coverage rate, normalized keyword coverage rate \(R_{N_i} = R_i / R_{\max}\) where \(R_i = r_i / l_i\), and Flesch reading-ease readability.
• Evaluation across seven LLMs spanning four closed-source frontier models (GPT-5, GPT-4.1, GPT-4o, Gemini-2.5-flash) and three open-source models (LLaMA-3.1-8B, Qwen3-8B, Gemma-3-4B).
• Obfuscated-code evaluation regime (identifier renaming, control-flow alteration, documentation stripping) to stress-test model semantic robustness.
• Six curated repositories (folder depth 7–13, 1116–1725 functions) released from CodeSearchNet, CodeXGLUE, and pybind.

Results

• +8% average SentenceBERT semantic similarity between parent- and child-folder summaries vs. both HR-CS and CS-BF baselines, across all folder depths and all seven LLMs.
• Up to +38% normalized keyword coverage rate over structured prompting baselines with certain LLMs.
• At function level, Agent4cs outperforms both baselines on all metrics; best single model: GPT-4.1 achieves Rouge-L 0.189, BLEU-1 0.197, BERTScore F1 0.826, SentenceBERT 0.672.
• GPT-5 consistently scores \(\geq 3.81\) on LLM-as-a-judge (1–4 scale) across all judge models; occasionally achieves 4.0.
• Obfuscated code shows only minimal performance degradation relative to clean code, indicating LLMs can extract functionality despite identifier renaming.
• Qwen3-8B underperforms on lexical/semantic metrics but matches frontier models on LLM-as-a-judge, attributed to distillation from teacher models aligning its outputs with GPT-style text.
• GPT-4.1 as judge exhibits leniency (favors code intelligence over language quality); GPT-5 as judge is stricter (compact models often score \(< 3.0\)).

Limitations

• Keyword extraction via TF-IDF is a lossy proxy for child-folder semantics; semantic nuances not captured by term frequency are silently dropped as context propagates upward.
• No folder-level ground truth exists; all hierarchical metrics are reference-free and measure internal consistency (parent-child similarity) rather than factual correctness against any oracle.
• LLM-as-a-judge intentionally excluded from hierarchical evaluation to avoid self-evaluation bias, limiting interpretability of hierarchical quality.
• Only six repositories evaluated; generalization to non-Python languages or shallow/wide repository structures is untested.
• The iterative refinement loop (summarization ↔ QA) has no stated convergence criterion or maximum iteration count documented in the excerpted text.
• API-dependent models (GPT-5, GPT-4.1, GPT-4o, Gemini-2.5-flash) are not reproducible without access; open-source alternatives lag on most metrics.

Relevance to Harnesses / Meta-Harnesses

Agent4cs is a canonical example of a task-decomposition harness: a meta-level controller that routes inputs through specialized sub-agents (summarize → extract → QA) in a fixed topological order over a data-structure (the folder tree), rather than having a single model handle everything. The keyword-extraction agent functions as a context compression module within the harness, solving the input-budget problem that meta-harnesses must address when aggregating outputs from many prior agent calls. The QA agent's iterative refinement loop is a self-repair sub-harness embedded within the outer bottom-up traversal, a pattern increasingly central to multi-agent meta-harness design. Compared to flat-prompting systems like Claude Code cited in the paper, this framework demonstrates that harness-imposed structure (role specialization, hierarchical traversal order, feedback loops) can yield measurable quality gains independent of underlying model capability.