Skip to content

Learning to Summarize by Learning to Quiz: Adversarial Agentic Collaboration for Long Document Summarization

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Adversarial agentic collaboration for long-document summarization [ICLR 2026]

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SummQ is an adversarial multi-agent framework for long document summarization that pairs a summarization pipeline with a quiz generation pipeline, using comprehension questions as an extrinsic quality signal to iteratively refine summaries. A dedicated Examinee agent validates whether the generated summary contains sufficient information to answer the quiz, closing the feedback loop. The framework significantly outperforms single-LLM prompting and prior multi-agent baselines on MENSA, BookSum, and GovReport.

Problem

LLMs processing long documents suffer from information loss, factual inconsistencies, and coherence degradation. Existing multi-agent summarization systems (e.g., HM-SR, C.MultiLLM) largely rely on self-verification, introducing systematic bias: a model that generated a flawed summary tends to miss the same flaws when reviewing it. There is no strong external signal to probe what is missing from a summary.

Method

SummQ operates as an iterative multi-agent harness with five specialized agent groups and two coupled task streams:

Summarization stream: \(G_s\) (Summary Generators), \(R_s\) (Summary Reviewers).
Quizzing stream: \(G_q\) (Quiz Generators), \(R_q\) (Quiz Reviewers).
Validation bridge: \(E\) (Examinee agent) attempts to answer quiz questions using only the current summary, producing feedback \(F_e\) split between both streams.

Each iteration \(t\) runs four stages: 1. \(S^{(t)} \leftarrow \text{GENERATE}(G_s, D, S^{(t-1)})\) and \(Q^{(t)} \leftarrow \text{GENERATE}(G_q, D, Q^{(t-1)})\) in parallel. 2. \(R_s\) reviews \(S^{(t)}\) against \(Q^{(t)}\) and \(D\); \(R_q\) reviews \(Q^{(t)}\). 3. \(E\) takes the quiz using \(S^{(t)}\) only; its feedback merges into \(F_s^{(t)}\) and \(F_q^{(t)}\). 4. If both feedback sets are empty, output \((S^{(t)}, Q^{(t)})\); otherwise loop up to \(T_{\text{iter}}\) times.

Generator collaboration (Algorithm 2, 4 phases): independent drafting → aggregation (an \(A_\text{Agg}\) agent merges drafts) → best-draft selection (a ranker \(A_\text{Ranker}\) picks \(z_\text{best}\)) → collective voting between \(z_\text{agg}\) and \(z_\text{best}\).

Reviewer collaboration (Algorithm 3, 4 phases): independent annotation → issue categorization (agreed: flagged by \(\geq 2\) reviewers; contested: flagged by \(< 2\)) → structured debate for contested issues with majority vote → final issue list \(I = M \cup K\).

Quiz generators produce 30 question-answer pairs per quiz (10 each: multiple-choice, true/false, short-answer). Default configuration: \(T_{\text{iter}} = 3\), \(T_{\text{debate}} = 1\), GPT-4O backbone, 3 agents per component in the COMBO variant.

Key Contributions

  • SummQ framework: adversarial coupling of summarization and quiz generation as mutual quality checks, mediated by an Examinee agent.
  • Dual collaboration protocols: distinct Generate (vote-based ensemble) and Review (debate + majority-vote) algorithms applicable to both task streams.
  • SotA on MENSA and BookSum: surpasses supervised fine-tuning, strong proprietary LLM prompting, and prior multi-agent baselines across ROUGE and BERTScore.
  • Ablation analyses: component-level ablations, iteration-count sweeps (\(T_{\text{iter}} \in \{1\ldots5\}\)), agent-count sweeps (1–5 per component), and backbone-LLM substitution experiments.

Results

  • MENSA (SummQ_COMBO): R-1 41.58, R-2 10.96–11.08, R-L 18.24, BSF1 62.76; best overall vs. all baselines.
  • BookSum (SummQ_COMBO): R-1 44.62, R-2 11.14, R-L 20.38, BSF1 61.49; most substantial gains, outperforms all baselines on all metrics.
  • GovReport (SummQ_COMBO): R-1 52.79, R-2 18.47, R-L 21.76, BSF1 65.46; outperforms all prompting baselines; some supervised fine-tuning models (U.FORMER, SLED, CACHED) remain competitive on specific metrics.
  • LLM-as-a-Judge (GPT-5 on MENSA): SummQ_COMBO wins 77–80% vs. GPT-4O and 66–74% vs. O3 across iterations.
  • Human evaluation (20 NLP papers, 5 PhD judges): SummQ_COMBO wins 88% vs. GPT-4O, 82% vs. O3; rephrased (length-debiased) variant SummQ_COMBO_R wins 65% and 60%, respectively.
  • Ablation – iterations: performance peaks at \(T_{\text{iter}} = 3\) (BSF1 62.76); gains plateau or reverse at \(T_{\text{iter}} \geq 4\).
  • Ablation – agent count: generally monotone improvement up to 4–5 agents; 5-agent R-1 reaches 42.52 on MENSA.
  • Ablation – component isolation: Summary Generators and Summary Reviewers show the largest benefit from multi-agent ensembling.

Limitations

  • Iterative refinement exhibits diminishing returns beyond 3 iterations; additional compute does not yield proportional quality gains.
  • On GovReport, large-scale supervised fine-tuning baselines (U.FORMER, SLED, CACHED) remain competitive or superior on select metrics, suggesting the framework does not fully close the gap with in-domain training.
  • Backbone is primarily GPT-4O; the text is truncated before reporting full open-source model results (DeepSeek-R1, Qwen3-32B ablations incomplete in the provided text).
  • Cost and latency are not reported; the ensemble-of-agents design is substantially more expensive than single-pass prompting.
  • Self-referential evaluation risk: LLM-as-a-Judge uses GPT-5, which may share biases with GPT-4O-based SummQ agents.

Relevance to Harnesses / Meta-Harnesses

SummQ is a concrete example of a domain-specific meta-harness: a fixed orchestration loop that coordinates heterogeneous specialized sub-agents (generators, reviewers, examinees) across two coupled task streams, with structured feedback aggregation and iterative refinement as first-class harness concerns. The Generate and Review protocols (Algorithms 2–3) are reusable harness primitives — vote-based ensemble generation and debate-based consensus review — that are task-agnostic and could be lifted into a general-purpose agent orchestration framework. The key architectural insight for harness designers is the adversarial auxiliary task pattern: coupling a primary task agent with a secondary agent whose objective is structurally adversarial (quiz generation pressures summary completeness), providing a quality signal that pure self-critique cannot. This pattern generalizes beyond summarization and is directly relevant to any harness that must validate LLM-generated artifacts without ground truth.