Skip to content

Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Tasks

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CGM integrates repository-level code graphs directly into open-source LLM attention mechanisms—via a text encoder, MLP adapter, and graph-aware attention mask—enabling agentless resolution of real-world GitHub issues. On SWE-bench Lite, CGM achieves 43.00% resolution rate using Qwen2.5-72B, ranking first among open-weight methods and surpassing the prior best open-source approach by 12.33 percentage points.

Problem

Repository-level software engineering (issue fixing, cross-file code completion) requires understanding both semantic content and structural dependencies across thousands of files. Existing open-source LLM approaches flatten code into linear sequences, losing inter-file dependency structure, while leading solutions depend on proprietary closed-source agents (GPT-4o, Claude 3.5 Sonnet) that introduce unpredictability, privacy concerns, and prevent task-specific fine-tuning.

Method

CGM represents each repository as a directed heterogeneous graph G=(V,E) with 7 node types (REPO, PKG, FILE, CLASS, FUNC, etc.) and 5 edge types (contains, calls, imports, extends), constructed via AST parsing + lightweight semantic analysis.

Semantic integration: Each node's text is encoded by a fine-tuned CodeT5+ encoder in 512-token chunks, then projected into the LLM decoder's embedding space via a two-layer MLP adapter (GELU activation), compressing each chunk to a single node token—effectively extending effective context by 512×.

Structural integration: The causal attention mask between node tokens is replaced by a graph-aware attention mask derived from the adjacency matrix, so attention flows only between graph-adjacent nodes, mimicking spatial GNN message passing.

Training: (1) Subgraph reconstruction pre-training (Graph-to-Code task on randomly sampled subgraphs); (2) noisy fine-tuning on SWE-bench issue–patch pairs with 10% irrelevant file injection and 10% oracle file omission for robustness. Encoder, adapter, and decoder are jointly fine-tuned with LoRA.

Graph RAG framework (4 steps): Rewriter (Extractor + Inferer) enriches the query → Retriever builds a connected subgraph via lexical+semantic search + one-hop expansion → Reranker selects Top-5 files in two stages → Reader (CGM) generates the patch using both graph node tokens and selected file text tokens.

Key Contributions

  • Novel CGM architecture integrating repository code graphs into decoder-only LLMs through semantic (encoder+adapter) and structural (adjacency-based attention mask) channels.
  • Subgraph reconstruction pre-training task (Graph-to-Code) enabling the model to jointly learn semantic and structural code representations.
  • 4-module agentless Graph RAG pipeline (Rewriter→Retriever→Reranker→Reader) that avoids multi-turn agent interaction while outperforming 10-step agentless baselines.
  • SOTA among open-weight models on SWE-bench Lite (43.00%), SWE-bench Verified (#1 open-weight), and SWE-bench-java Verified (14.29%, vs. 9.89% for best SWE-Agent variant).

Results

  • SWE-bench Lite: 43.00% resolution rate (CGM-SWE-PY, Qwen2.5-72B); #1 open-weight, #2 open-source-system, #8 overall; +12.33% over Moatless+DeepSeek-V3 (30.67%); +2.33% over Agentless+Claude-3.5-Sonnet (40.67%).
  • SWE-bench Verified: 50.40% (Pass@1), #1 open-weight, #5 open-source-system.
  • SWE-bench-java Verified: 14.29% (CGM-Multi); best competitor SWE-Agent+DeepSeek-Chat-V2 at 9.89%.
  • CrossCodeEval (code completion): CGM-Multi-72B achieves EM 51.61 / ES 84.62 (Java), EM 46.23 / ES 82.16 (Python), outperforming Mixtral-123B, DeepSeek-236B, and standalone Qwen2.5-72B.
  • ComplexCodeEval: CGM-Multi largely outperforms all baselines, e.g., Java EM 47.00 vs. 37.00 (next best).
  • Test-time scaling: Pass@3 yields 46.67% on SWE-bench Lite and 53.20% on SWE-bench Verified.
  • Ablation (Table 5): Removing all RAG modules (W/O R3) drops to 9.67%; removing CGM Reader in favor of flat graph drops to 5.33%.
  • Ablation (Table 6): Graph-aware attention mask (+2.9 EM Java) and reconstruction pre-training (+8.83 EM Java) both significant contributors.

Limitations

  • Evaluated primarily on Python (SWE-bench Lite/Verified); Java coverage limited to one smaller benchmark (91 issues).
  • CGM-Multi (multi-task, multi-language) trails CGM-SWE-PY by 6.33% on Lite due to task/language breadth trade-off.
  • Code graph construction and subgraph retrieval add preprocessing overhead not present in pure prompt-based methods (see Appendix C).
  • 7B backbone (Qwen2.5-Coder-7B) achieves only 4.00% on SWE-bench Lite, indicating the approach is sensitive to base model capacity at small scales.
  • Context length for very large repositories is managed by subgraph sampling; extremely large codebases may require aggressive pruning that risks dropping relevant context.
  • No comparison against the most recent frontier agents (e.g., post-May 2025 systems) in the main evaluation.

Relevance to Agentic AI / LLM Agents

CGM directly challenges the prevailing assumption that repository-level SE tasks require multi-turn LLM agents with tool use, showing that an agentless, graph-structured single-pass architecture can match or exceed agent-based systems while using open-source models—a significant finding for the agent design space. The work demonstrates that structural context injection (replacing attention masks with graph adjacency) is a viable alternative to iterative tool-use loops, with fewer compounding errors and better auditability. For researchers tracking LLM agents for software engineering, CGM establishes a strong non-agent baseline and a new modality-alignment paradigm (graph ↔ LLM) that complements retrieval-augmented agent architectures. The 4-step Graph RAG pipeline also offers a reusable template for grounding agentic or agentless SE systems in explicit dependency structure rather than linearized code.