Skip to content

A Stepwise Questioning Expert-Editor Multi-Agent Framework for Long-Document Summarization

🕒 Published (v1): 2026-07-11 16:37 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SQ2E proposes a four-role multi-agent pipeline (Expert, Editor, Author, Evaluator) for long-document summarization where Expert and Editor agents issue stepwise questions to an Author agent, driving iterative summary refinement. An Evaluator gate accepts or rejects each revision pass. Experiments on arXiv and PubMed with three LLMs show consistent ROUGE and BERTScore gains over direct generation and the HERA baseline for smaller models.

Problem

LLM-based long-document summarization suffers from information loss, hallucination, and factual inconsistency when documents exceed context limits; prior segment-and-aggregate approaches improve coverage but do not actively surface missing key information or correct surface-level errors post-generation.

Method

SQ2E (Stepwise Questioning Expert-Editor) orchestrates four roles in a pipeline: 1. Writer segments the document by section, generates local summaries, then aggregates an initial global summary. 2. Expert (semantic focus) formulates a context-specific question plan (≤5 questions per round) drawn from a curated bank targeting completeness, coherence, relevance, and factual consistency; the Author answers each question and revises the summary. 3. Editor (surface/linguistic focus) issues up to 4 yes/no questions comparing the summary against the source; Author revises only on "yes" responses. 4. Evaluator compares the previous and revised summary on the four criteria and selects the better version after every revision cycle, preventing degradation.

Communication follows a strict pipeline topology; Expert questioning precedes Editor questioning. The question bank is fixed and manually designed; the Expert and Editor select applicable questions per document context.

Key Contributions

  • Four-role multi-agent pipeline (Expert, Editor, Author, Evaluator) that separates semantic and linguistic refinement into distinct, sequentially staged questioning agents.
  • A curated question bank covering completeness, coherence, relevance, and factual consistency, with empirical analysis of which questions produce valid refinements.
  • Evaluator-gated refinement preventing regressions: each revision is accepted only if it outperforms the prior version.
  • Experiments on both commercial (DeepSeek-R1) and open-source (LLaMA-3.1-8B/70B) models on arXiv and PubMed.

Results

Baselines: DG (Direct Generation), HERA (Li et al., 2025a). Test set: 300 instances per dataset. Metrics: ROUGE-1/2/L, BERTScore (BS), FactCC (FC).

arXiv: - LLaMA-3.1-8B: SQ2E R-1=34.19 / R-2=8.07 / R-L=16.89 / BS=81.38 / FC=60.67 vs. DG (32.22/8.23/16.25/80.41/54.33) - LLaMA-3.1-70B: SQ2E R-1=38.89 / R-2=11.27 / R-L=18.95 / BS=84.01 / FC=50.67 vs. DG (33.53/9.44/17.65/80.99/61.00); FC drops below DG - DeepSeek-R1: SQ2E R-1=38.94 / R-2=11.74 / R-L=20.22 / BS=82.33 / FC=52.67 vs. DG (36.19/10.64/17.88/81.28/46.00)

PubMed: - LLaMA-3.1-8B: SQ2E R-1=34.65 / R-2=9.16 / R-L=17.35 / BS=82.41 / FC=55.00 vs. DG (32.22/8.23/16.25/80.41/54.33) - LLaMA-3.1-70B: SQ2E underperforms DG on R-1/R-2/R-L (DG: 39.58/14.25/20.70; SQ2E: 36.82/12.28/20.60) - DeepSeek-R1: SQ2E R-1=37.29 / R-2=12.04 / R-L=19.86 / BS=83.21 vs. DG (35.43/11.03/17.70/81.82) - Expert questions yield 3–4 valid refinements per document on average; Editor questions yield 2–3.

Limitations

  • Performance degrades or reverses on LLaMA-3.1-70B across multiple metrics and both datasets; authors attribute this to precision loss under their A100 inference setup, but the root cause is not rigorously established.
  • Only scientific document datasets (arXiv, PubMed) are evaluated; generalization to news, legal, or other long-document domains is untested.
  • No human evaluation; reliance on ROUGE/BERTScore/FactCC only.
  • Test set capped at 300 sampled instances per dataset; statistical significance is not reported.
  • Question bank is manually crafted; coverage and transferability to new domains require additional design effort.
  • Multi-pass questioning multiplies LLM inference cost substantially; wall-clock and token costs are not reported.

Relevance to Harnesses / Meta-Harnesses

SQ2E is a concrete example of a role-structured, evaluator-gated multi-agent harness: the fixed pipeline topology (Writer → Expert-QA loop → Editor-QA loop → Evaluator selection) mirrors meta-harness design patterns where a controller orchestrates specialized sub-agents through defined communication channels and quality gates prevent output regression. The Evaluator gate in particular implements an inline judge that short-circuits bad refinements—directly analogous to verification or critic nodes in general-purpose agentic harnesses. The manually designed question bank functions as a domain-specific prompt scaffold baked into the harness, illustrating how task-specific knowledge can be hard-coded into an otherwise generic iterative-refinement topology. For researchers tracking harness design, this paper provides an empirical case study in how communication policy (pipeline vs. debate vs. parallel), role specialization, and gating interact with underlying model capability.