Skip to content

Learning to Summarize by Learning to Quiz: Adversarial Agentic Collaboration for Long Document Summarization

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Adversarial quiz-generator agent guides summarizer; novel self-improving document agent loop

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SUMMQ is an adversarial multi-agent framework for long document summarization that pairs a summarization pipeline with a quizzing pipeline, using comprehension questions as a continuous quality signal. An examinee agent attempts to answer quiz questions using only the generated summary, producing feedback that drives iterative refinement. The system achieves state-of-the-art ROUGE and BERTScore on MENSA and BookSum, and is preferred by human judges at 88% over GPT-4o and 82% over o3.

Problem

LLMs struggle with long document summarization due to information loss, factual inconsistencies, and coherence degradation over extended contexts. Existing multi-agent summarization approaches rely on self-verification, introducing self-consistency biases and missing errors that an independent adversarial check would catch.

Method

SUMMQ runs two parallel pipelines—summarization and quiz generation—through \(T_\text{iter}\) iterations. Each iteration proceeds as follows:

Generator collaboration (Algorithm 2): \(n\) generator agents independently draft candidates, an aggregator merges them into \(z_\text{agg}\), a ranker selects the best individual draft \(z_\text{best}\), and all generators vote between \(\{z_\text{agg}, z_\text{best}\}\).

Reviewer collaboration (Algorithm 3): \(n\) reviewer agents independently annotate the candidate. Issues flagged by \(\geq 2\) reviewers are "agreed" issues \(M\); those with \(<2\) flags are "contested" \(C\). Contested issues undergo \(T_\text{debate}\) rounds of structured debate followed by majority vote, yielding valid contested issues \(K\). The final issue list is \(I = M \cup K\).

Examinee agent: Given quiz \(Q^{(t)}\) and summary \(S^{(t)}\), the examinee attempts to answer each question using only the summary. Failures generate feedback that is merged into both the summary and quiz feedback streams.

The quiz consists of 30 question-answer pairs per document (10 each: multiple-choice, true-false, short-answer). Iteration terminates when \(F_s^{(t)} = \emptyset\) and \(F_q^{(t)} = \emptyset\), or at \(T_\text{iter}\) (default 3). Two configurations are evaluated: SUMMQ\(_\text{SOLO}\) (1 agent per component) and SUMMQ\(_\text{COMBO}\) (3 agents per component, default backbone GPT-4o).

Key Contributions

  • SUMMQ: an adversarial multi-agent framework coupling summarization and quiz generation as mutually constraining tasks.
  • Structured generator collaboration (independent draft → aggregation → best selection → collective voting) and reviewer collaboration (independent annotation → issue categorization → debate → majority vote).
  • Examinee agent that provides cross-pipeline feedback by attempting quiz answering from the summary alone.
  • State-of-the-art results on MENSA and BookSum across ROUGE, BERTScore, LLM-as-a-Judge (GPT-5), and human evaluation; ablations over component count, iteration count, and agent backbone.

Results

  • MENSA: SUMMQ\(_\text{COMBO}\) achieves R-1 = 41.58, R-2 = 11.08 (vs. GPT-5 baseline R-1 = 37.38), BSF1 = 62.76; outperforms all prompting and multi-agent baselines.
  • BookSum: SUMMQ\(_\text{COMBO}\) R-1 = 44.62, R-2 = 11.14, BSF1 = 61.49; substantial margins over all baselines including HM-SR and C.MultiLLM.
  • GovReport: SUMMQ\(_\text{COMBO}\) R-1 = 52.79, BSF1 = 65.46; outperforms all prompting baselines; some supervised fine-tuning methods (UFOMER, SLED, CACHED) remain competitive on specific metrics.
  • LLM-as-a-Judge (GPT-5, MENSA, iteration 3): SUMMQ\(_\text{COMBO}\) wins 80% vs. GPT-4o, 74% vs. o3.
  • Human evaluation: SUMMQ\(_\text{COMBO}\) wins 88% vs. GPT-4o, 82% vs. o3; length-controlled SUMMQ\(_\text{COMBO}\)R still wins 65% vs. GPT-4o, 60% vs. o3.
  • Ablations: performance peaks at \(T_\text{iter} = 3\); gains diminish or reverse at \(T_\text{iter} \geq 4\). All four components benefit from multi-agent ensemble, with Summary Generators and Summary Reviewers contributing most. More agents (up to 5) show further ROUGE gains.
  • Backbone generalization: SUMMQ improves GPT-4o-mini, GPT-4.1, o3, DeepSeek-R1 baselines substantially (e.g., GPT-4.1 baseline R-1 = 30.31 → SUMMQ R-1 = 49.17 on MENSA).

Limitations

  • Significant computational cost: SUMMQ\(_\text{COMBO}\) with 3 agents per component and 3 iterations multiplies inference calls substantially; not analyzed in the paper.
  • Diminishing returns beyond 3 iterations with no principled stopping criterion beyond the no-feedback condition or the iteration cap.
  • GovReport shows no clear win over domain-specialized supervised fine-tuning baselines, suggesting the framework may be less effective when reference summaries are very short or stylistically constrained.
  • Quiz quality itself is unverified by an external ground truth—quiz generators can fail to cover all important content, and this failure would not be caught by the framework.
  • Evaluation uses GPT-5 as judge and also as a baseline; this creates potential circular evaluation issues not explicitly addressed.

Relevance to Agentic AI / LLM Agents

SUMMQ is a concrete instantiation of adversarial multi-agent collaboration where two heterogeneous agent roles (summarizer and quizzer) exert mutual pressure on each other's outputs, a pattern directly relevant to building self-correcting agent systems without human-in-the-loop feedback. The examinee agent acts as an automated critic that grounds feedback in verifiable task performance rather than open-ended self-critique, offering a template for quality-gated iterative refinement in long-horizon agentic tasks. The structured debate mechanism for contested issues is a practical implementation of multi-agent deliberation with majority-vote resolution, applicable beyond summarization to any agent pipeline requiring consensus on ambiguous quality signals. The framework demonstrates that task decomposition into complementary adversarial subtasks (generate vs. challenge) is an effective architecture for improving output quality in domains where self-consistency bias limits single-agent or self-review approaches.