Skip to content

RoadMapper: A Multi-Agent System for Roadmap Generation of Solving Complex Research Problems

🕒 Published (v1): 2026-04-30 09:08 UTC · Source: Arxiv · Venue: ACL 2026 · link

Why this paper was selected

Multi-agent roadmap generation guides complex research problem solving; ACL 2026

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RoadMapper is a six-agent LLM system for automatically generating hierarchical research roadmaps from complex research problems. It introduces the RoadMap benchmark (1,705 expert-annotated roadmaps across 10 fields, bilingual) and a DPO-trained evaluator agent that drives an iterative critique-revise loop. The system improves baseline LLM performance by >8% on average and reduces expert design time by 84%.

Problem

Existing structured content generation work (mindmaps, flow graphs, knowledge graphs) addresses shallow extraction tasks and lacks professional depth, multi-field coverage, and step-by-step guidance for solving complex research problems. No prior benchmark or method targets the roadmap generation task—producing a logically ordered, hierarchically decomposed action plan \(F: x_{\text{problem}} \to y_{\text{roadmap}}\)—and direct LLM prompting fails on three axes: insufficient professional knowledge, unreasonable task decomposition granularity, and disordered logical relationships between subtask nodes.

Method

RoadMapper decomposes roadmap generation into three sequential stages handled by six specialized LLM agents:

  1. Init Agent (I): generates an initial roadmap draft from the research problem.
  2. Knowledge Agent (K): retrieves the top-\(K\) skill points from Skill-Repo via vector similarity and augments the draft: \(y_0 = K(\text{knowledge}, y_{\text{initial}})\).
  3. Iterative critique-revise-evaluate loop:
  4. Logic Critique Agent (L): flags violations of parent-child and sibling logical relationships, outputs revision suggestions \(LC_t\).
  5. Granularity Critique Agent (G): flags nodes that are too detailed (over-split) or too brief (under-split), outputs \(GC_t\).
  6. Revise Agent (R): applies both critique outputs: \(y_{t+1} = R(y_t, LC_t, GC_t)\).
  7. Evaluate Agent (E): scores \(y_{t+1}\); if score \(\geq 80\) (passing threshold), outputs the roadmap; otherwise increments \(t\) up to \(T_{\max}=5\).

The Evaluate Agent is fine-tuned with DPO on 818 roadmaps. Preference pairs are constructed by having Qwen3-32B generate 10 evaluation candidates per roadmap, then seven experts vote; the top-voted candidate is \(y_w\) and the second-ranked is \(y_l\) (chosen to force fine-grained discrimination). The DPO objective is:

\[\mathcal{L}_{\text{DPO}} = -\mathbb{E}_D\left[\log\sigma\!\left(\beta\log\frac{\pi_\theta(y_w|x)/\pi_{\text{ref}}(y_w|x)}{\pi_\theta(y_l|x)/\pi_{\text{ref}}(y_l|x)}\right)\right]\]

Roadmaps are represented as Markdown with heading-level hierarchy, numeric indices (e.g., 1.3.2), and bracketed titles.

The RoadMap benchmark includes 1,705 expert-annotated golden roadmaps (500 EN / 1,205 CN) derived from ProQuest/CNKI graduate dissertations, plus 8,436 skill points. Four metrics are defined: DegreeScore (out-degree fidelity), DepthScore (depth fidelity), StepScore (key-step coverage, GPT-4o mini judged), and LogicScore (logical coherence, GPT-4o mini judged).

Key Contributions

  • RoadMap benchmark: 1,705 bilingual expert roadmaps across 10 research fields and 5 problem types, with an 8,436-item Skill-Repo providing professional knowledge.
  • RoadMapper system: six-agent pipeline with DPO-aligned evaluator enabling early stopping, averaging 1.51–1.64 critique-revise iterations in practice.
  • Four novel evaluation metrics: DegreeScore, DepthScore, StepScore, LogicScore covering both structural fidelity and content quality.
  • DPO preference dataset: 818 roadmaps with second-best negative sampling to elicit fine-grained expert alignment in the Evaluate agent.

Results

  • RoadMapper achieves SOTA across all 11 tested base models on both English and Chinese splits.
  • vs. Direct Prompting: +7.53 / +9.16 average improvement for Llama 3.3 70B (EN/CN); +8.11 / +6.88 for DeepSeek-V3.2.
  • vs. ReConcile (round-table multi-agent): +4.20 average for Llama 3.3 70B.
  • vs. DyLAN (dynamic multi-agent): consistent improvement across all models and metrics.
  • Ablation on Llama 3.3 70B: removing K agent drops StepScore by 5.84 (overall −3.72); removing L agent drops LogicScore by 5.43; DPO ablation costs −2.21 average; merging L+G into one agent degrades all metrics.
  • Early stopping: average 1.64 iterations (Llama 3.3 70B) and 1.51 (GPT-4o mini), substantially below \(T_{\max}=5\).
  • 84% reduction in roadmap design time compared to human experts.

Limitations

  • Evaluation of content metrics (StepScore, LogicScore) relies on GPT-4o mini as judge, inheriting that model's biases and potential inconsistencies.
  • DPO backbone requires \(\geq\)14B parameters; Qwen3-8B with DPO underperforms the no-DPO baseline, limiting accessibility.
  • Skill-Repo is domain-specific (10 fields, dissertation-sourced); coverage of rapidly emerging or interdisciplinary areas may be insufficient.
  • The benchmark is drawn exclusively from ProQuest/CNKI graduate dissertations, which may not generalize to applied industrial research or non-academic problem formulations.
  • Roadmap quality and the passing threshold of 80 are set heuristically; the stopping criterion's sensitivity is not formally analyzed.

Relevance to Agentic AI / LLM Agents

RoadMapper is a concrete example of task-decomposed multi-agent collaboration where distinct agents handle knowledge retrieval, structural critique, logical critique, revision, and DPO-aligned self-evaluation—demonstrating that role specialization outperforms generic multi-agent debate (ReConcile) and dynamic selection (DyLAN) on a structured generation task. The DPO-trained evaluator acting as an early-stopping gatekeeper is a practical pattern for autonomous quality control without human-in-the-loop review at inference time. The iterative critique-revise-evaluate loop directly instantiates the "reflection" paradigm central to modern agentic frameworks, and the comparison with CoT, Best-of-N, ReConcile, and DyLAN provides a rare controlled study of prompting vs. multi-agent strategies on a non-trivial structured output task.