Skip to content

RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RepoMaster is an autonomous agent framework that reuses existing GitHub repositories—rather than generating code from scratch—to solve complex real-world tasks. It builds static structural graphs (HCT, FCG, MDG) and an importance-scored core-component index to feed only the essential code into the LLM's context window, then explores and executes iteratively. On GitTaskBench it raises task-pass rate from 40.7% to 62.9% while cutting token usage by 95%.

Problem

Existing code agents (OpenHands, SWE-Agent) lack principled mechanisms for navigating large repositories: they either flood the LLM context with irrelevant code or rely on incomplete README files, leading to low success rates and high token costs. Purely generative approaches cannot scale to full repository-level tasks. GitHub's 28M+ public repositories remain underutilized as modular components.

Method

RepoMaster operates in three stages:

  1. Repository Search — extracts key entities from the user's natural-language task, queries GitHub, scores candidates by README quality and star count, and returns the top-3 as structured JSON.

  2. Hierarchical Repository Analysis — performs a single AST walk over all source files to construct three artefacts:

  3. Hierarchical Code Tree (HCT) \(T\): package → module → class → function containment map.
  4. Function Call Graph (FCG) \(G_f = (V_f, E_f, w_f)\): directed weighted call graph over functions.
  5. Module Dependency Graph (MDG) \(G_m = (V_m, E_m, w_m)\): explicit import-dependency graph over modules.

Each module \(m\) is scored \(I(m) = \min\!\bigl(\sum_{i=1}^{6} w_i s_i(m),\,10\bigr)\) over six features: MDG centrality (personalized PageRank), cyclomatic complexity, import/call frequency, semantic keywords, docstring richness, and Git commit recency (equal weights, \(w_i \equiv 1\)). Class importance is \(J(c) = I(\mu(c)) + |F_c|/\max|F_{c'}| + \text{Calls}(F_c)/\max\text{Calls}(F_{c'})\). Top-\(k\) classes become the core components forming the initial LLM context.

  1. Autonomous Exploration & Execution — the agent iteratively invokes three tool categories (Granular Code View via HCT, Dependency Analysis via FCG/MDG, Search via keyword matching) and interleaves code writing, execution, and feedback-driven re-exploration. A multi-level information selection strategy activates when context exceeds \(L = 8000\) tokens: AST subtree extraction for code, chunk-rank-retain (chunk size \(L_c=1000\), keep top \(n_c=4\)) for documents, and head+tail retention for logs.

Key Contributions

  • End-to-end agent framework for reusing GitHub repositories to solve complex tasks without generating from scratch.
  • Hybrid structural modeling: HCT + FCG + MDG providing a deterministic, loss-minimal repository synopsis.
  • Importance scoring (module-level PageRank-based + class-level refinement) to select core components for the initial LLM context.
  • Multi-level information selection (AST pruning, chunk ranking, log truncation) that keeps each interaction under the token budget.
  • GitTaskBench: a new benchmark of 54 tasks across 18 repositories spanning image processing, video, speech, office automation, and security.

Results

  • MLE-R (22 Kaggle-derived tasks):
  • RepoMaster + Claude 3.5: 95.45% valid submission rate, 27.27% any-medal rate (22.73% gold) vs. OpenHands + Claude 3.5: 45.45% valid submissions, 4.55% any-medal — >5× improvement in medal rate.
  • RepoMaster + GPT-4o: 86.36% valid submissions, 18.18% any-medal.
  • GitTaskBench (54 tasks, 18 repos):
  • RepoMaster + Claude 3.5: 75.92% execution completion, 62.96% task pass rate vs. OpenHands (53.70%, 40.74%) and SWE-Agent (41.67%, 22.23%).
  • Token consumption: 154k vs. 2,883k tokens/task (OpenHands) — ~95% reduction.
  • RepoMaster + DeepSeek V3: 62.96% pass rate; + GPT-4o: competitive, maintaining significant margins over baselines.

Limitations

  • Evaluation restricted to Python repositories (AST-based analysis is language-specific).
  • Repository search step assumes suitable public repositories exist and are discoverable; performance degrades when no strong candidate is found.
  • Context budget \(L=8000\) tokens per interaction may still be insufficient for very large, deeply coupled repositories.
  • Importance scoring uses equal feature weights (\(w_i \equiv 1\)); sensitivity analysis is deferred to an appendix but the scheme may not generalize across all repository types.
  • GitTaskBench covers only 18 repositories and 54 tasks; broader generalization is not yet established.
  • No evaluation of multi-repository composition tasks.

Relevance to Harnesses / Meta-Harnesses

RepoMaster is a task-solving meta-harness that orchestrates repository search, static graph construction, importance scoring, and a dynamic exploration-execution loop as discrete, composable stages — precisely the architecture pattern meta-harnesses use to wrap lower-level tools. The three-stage pipeline (search → analyze → explore/execute) mirrors harness scaffolding where each stage feeds structured context forward rather than passing raw environment state. Its information-selection layer — deciding what to expose to the LLM at each step based on computed relevance — is directly analogous to context-management strategies in agent harnesses that must operate within fixed token budgets across many turns. The 95% token reduction result quantifies the payoff of harness-level context governance, providing a strong empirical data point for harness designers prioritizing efficiency.