Skip to content

RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RepoMaster is an autonomous agent framework that enables LLMs to solve complex real-world tasks by intelligently exploring and reusing GitHub repositories rather than generating code from scratch. It constructs static structural graphs (function-call, module-dependency, hierarchical code trees) to identify core components, then dynamically navigates the codebase with information pruning to stay within LLM context limits. On two benchmarks it substantially outperforms OpenHands and SWE-Agent while reducing token consumption by up to 95%.

Problem

Existing code agents (OpenHands, SWE-Agent) treat GitHub repositories as opaque blobs navigated via README files and simple find/edit commands. This fails for large, complex codebases because: (1) repositories have intricate structural dependencies across many files, and (2) naively loading repository content quickly exhausts LLM context windows. The result is poor task completion and high token cost when real-world tasks require deep repository reuse rather than generating code from scratch.

Method

RepoMaster operates in three stages:

1. Repository Search. Intent-driven deep search over GitHub using key entity extraction, README analysis, and star counts; top-3 candidates validated and returned as structured JSON.

2. Hierarchical Repository Analysis. A single AST walk over all source files produces three complementary structures: - Hierarchical Code Tree (HCT): package → module → class → function containment map. - Function Call Graph (FCG): weighted directed graph of function invocations. - Module Dependency Graph (MDG): weighted directed graph of import relations.

Core components are identified via a six-feature importance score per module (Dependency via personalized PageRank, Complexity, Usage, Semantic, Doc, Git), then class-level scores refine selection to top-k classes. An initial context is assembled from: full README, natural-language summaries of top-20 modules, source code of top-10 core classes, and path lists for remaining modules.

3. Autonomous Exploration & Execution. Three exploration tool categories (Granular Code View via HCT, Dependency Analysis via FCG/MDG, Keyword Search) let the agent navigate on-demand. Execution and exploration interleave continuously. Multi-level information selection activates when content exceeds the 8k-token per-interaction limit: AST subtree extraction for code, chunk ranking for documents, and head/tail retention for logs.

The system is built on AutoGen as the multi-agent dialog platform.

Key Contributions

  • Novel end-to-end framework for repository-based complex task solving using "reuse and adapt" rather than from-scratch generation.
  • Hybrid structural repository mapping combining HCT, FCG, and MDG with a principled six-feature importance scoring scheme for core component identification.
  • Context-aware exploration toolset enabling hierarchical, graph-guided codebase traversal within tight LLM context budgets.
  • Multi-level information selection strategy (AST extraction, chunk ranking, log head/tail) to maintain high signal-to-noise ratio across multi-turn interactions.
  • GitTaskBench: a new benchmark of 54 tasks across 18 repositories spanning image processing, video analysis, speech, physiological signals, office automation, and security.

Results

MLE-R (22 Kaggle-derived ML engineering tasks): - RepoMaster + Claude 3.5: 95.45% valid submission rate, 27.27% any-medal rate (22.73% gold) vs. best baseline OpenHands + Claude 3.5 at 45.45% valid, 4.55% any-medal — >5× improvement in medal rate. - RepoMaster + GPT-4o: 86.36% valid submission, 18.18% any-medal. - SWE-Agent and OpenHands: 0% medals with GPT-4o; ≤4.55% with Claude 3.5.

GitTaskBench (54 tasks, 18 repos): - RepoMaster + Claude 3.5: 75.92% execution completion, 62.96% task pass rate vs. OpenHands (53.70%, 40.74%) and SWE-Agent (41.67%, 22.23%). - Token consumption: RepoMaster 154k vs. OpenHands 2,883k vs. SWE-Agent 456k per task (~95% reduction vs. OpenHands). - RepoMaster + GPT-4o: 48.14% completion, 40.74% pass rate at 250k tokens vs. OpenHands 21.30%, 14.82% at 765k tokens.

Ablation (GPT-4o on GitTaskBench): - Removing all three mechanisms: pass rate drops from 40.74% → 24.07% (−16.67%); tokens drop to 112k (agent gives up early). - Removing code exploration alone: −5.56% pass rate, +13.6% tokens — most critical single component. - Removing hybrid hierarchical analysis or information selection individually: ~−1.86% pass rate each.

Limitations

  • Evaluated only on Python repositories; generalization to other languages (Java, C++, etc.) is not validated empirically.
  • Repository search relies on GitHub availability and README/star-count heuristics, which can fail for niche or poorly documented repos.
  • The importance scoring uses equal feature weights (wi ≡ 1) validated only through sensitivity analysis; optimal weighting may vary by domain.
  • GitTaskBench is a new, relatively small benchmark (54 tasks, 18 repos) introduced by the same authors, raising potential evaluation bias.
  • The 8k-token per-interaction limit and fixed hyperparameters (top-20 modules, k=10 classes) are not adaptively tuned to repository size or task complexity.
  • No multi-repository orchestration is demonstrated despite being cited as future work.

Relevance to Agentic AI / LLM Agents

RepoMaster advances the "tools-as-repositories" paradigm — treating GitHub as a dynamic tool library rather than requiring agents to generate complex code from scratch — which is a natural next step beyond single-function tool use. The core insight that static structural analysis (graph construction + importance scoring) should precede dynamic agentic execution is broadly applicable to any agent that must reason over large, structured external artifacts within finite context budgets. The 95% token reduction while achieving higher task pass rates demonstrates that principled information selection is a first-class concern for long-horizon agentic tasks, directly relevant to scaling code agents to real-world engineering workflows. The interleaved explore-execute loop and feedback-driven context management offer a reusable architectural template for agents operating on large knowledge bases beyond code.