Skip to content

HiRAS: A Hierarchical Multi-Agent Framework for Paper-to-Code Generation and Execution

🕒 Published (v1): 2026-04-20 02:59 UTC · Source: Arxiv · link

Why this paper was selected

Hierarchical multi-agent with adaptive pipelines for paper-to-code generation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HiRAS introduces a hierarchical multi-agent system for end-to-end automated reproduction of computational experiments from research papers, using proactive manager agents that actively supervise, inspect intermediate artefacts, and re-invoke subordinate agents to correct errors. It achieves 64.1% on PaperBench-CodeDev with Claude-Sonnet (>10% relative gain over prior SOTA) and consistently outperforms sequential pipeline baselines. The paper also exposes evaluator-hallucination failures in the Paper2Code benchmark's reference-free metric and proposes a corrected protocol (P2C-Ex).

Problem

Existing paper-to-code systems rely on fixed sequential agent pipelines with no global supervision: errors introduced in early phases (planning, analysis) propagate unchecked through subsequent coding and execution phases, leading to stalled workflows and incomplete repositories. Additionally, the reference-free evaluation protocol for Paper2Code systematically overestimates repository quality (an empty repository scores 3.89/5, config-only repositories score 4.67/5) due to LLM-judge hallucination.

Method

HiRAS decomposes reproduction into four primary phases—planning (split further into overall planning, architecture design, dependency modelling, and configuration generation), analysis, coding, and execution—each assigned to a dedicated specialised agent \(A_t = (\text{LLM}, \text{Mem}_t, T_t, E, K)\) operating via the ReAct paradigm with up to \(K\) reasoning-action iterations over a shared file-system workspace.

Two hierarchical manager agents are layered on top: - Planning manager \(A_\text{plan}\) oversees all four planning sub-agents. - Global manager \(A_\text{global}\) supervises \(\{A_\text{plan}, A_\text{analysis}, A_\text{code}, A_\text{exec}\}\).

Managers are not passive routers: they inspect workspace artefacts after each phase via file-system tools, diagnose failures, and call invoke(A_\text{sub}, p) to re-execute or correct specific subordinate agents. The entire workflow is a tree-structured recursive invoke call initiated by invoke(A_\text{global}, p_0), with agents able to call end_task(Report) to signal completion. The framework is scaffolded on smolagents.

P2C-Ex corrects Paper2Code evaluation by augmenting the evaluator prompt with (i) total file count, (ii) hierarchical repository structure, and (iii) anti-hallucination instructions.

Key Contributions

  • HiRAS framework: hierarchical multi-agent architecture with proactive supervisory manager agents for automated experiment reproduction, first of its kind in this domain.
  • State-of-the-art on PaperBench: 64.1% on CodeDev subset with Claude-Sonnet, surpassing all prior methods including those using proprietary LLMs; 57.4% with open-source DeepSeek-v3.1 exceeds previous SOTA using closed models.
  • P2C-Ex evaluation protocol: reference-free metric for Paper2Code that incorporates repository-level structural context, raising Pearson correlation with reference-based scores from \(r = 0.423\) (original) to \(r = 0.862\).
  • Ablation characterising the hierarchy's value: hierarchical supervision alone contributes ~10% absolute gain over specialised agents + iterative refinement.

Results

  • PaperBench-CodeDev (o3-mini-high evaluator):
  • HiRAS + Claude-Sonnet: 64.1% vs. PaperCoder 51.1% (prior SOTA, same backbone)
  • HiRAS + DeepSeek-v3.1: 57.4% vs. AutoReproduce 41.5% (+38.3% relative)
  • HiRAS + Qwen3: 45.7% vs. PaperCoder 36.9% (+23.8% relative)
  • Full PaperBench (ChatGPT-4o-mini evaluator):
  • HiRAS + Qwen3: 19.1% vs. baselines 11.2–11.5%
  • HiRAS + DeepSeek-v3.1: 21.5% vs. baselines 16.9–17.1%
  • Paper2Code (reference-based, Qwen3):
  • HiRAS: 2.45 vs. PaperCoder 1.83, AutoReproduce 2.45
  • Paper2Code (reference-based, DeepSeek-v3.1):
  • HiRAS: 3.64 vs. AutoReproduce 2.51
  • P2C-Ex Pearson correlation with reference-based: \(r = 0.862\) vs. original Ref-Free \(r = 0.423\)

Limitations

  • Evaluation on PaperBench covers only 20 ICML 2024 ML papers; generalisability to other domains or paper styles is unverified.
  • All experiments run on 8Ă— NVIDIA H200; computational cost of hierarchical re-invocation loops is not quantified in the text.
  • Manager agent prompts and coordination logic remain hand-crafted per phase; no mechanism for automatic adaptation to novel paper structures.
  • P2C-Ex still relies on LLM-as-judge (o3-mini-high); the reduced-but-nonzero hallucination risk is acknowledged but not fully resolved.
  • Paper2Code reference-based evaluation requires gold repositories, limiting scalability to papers without released code.

Relevance to Harnesses / Meta-Harnesses

HiRAS is a direct example of a meta-harness: a control layer that orchestrates a pipeline of specialised agents rather than executing tasks itself, with explicit looping, error detection, and corrective re-invocation logic forming the harness logic. The two-level hierarchy (planning manager + global manager) mirrors the pattern of a meta-harness wrapping domain-specific sub-harnesses, and Algorithm 1's recursive invoke function is essentially a harness scheduler with retry semantics. The P2C-Ex contribution is notable for the harness-tracking community because it highlights how evaluation harnesses themselves can be corrupted by LLM-judge hallucination—a systemic risk for any meta-harness that uses an LLM to judge intermediate artefact quality.