Skip to content

Select, Read, and Write: A Multi-Agent Framework of Full-Text-based Related Work Generation

🕒 Published (v1): 2025-05-26 08:02 UTC · Source: Arxiv · Venue: ACL 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes a three-agent framework—Selector, Reader, Writer—for generating academic related work sections by iteratively reading full-text reference papers and maintaining a bounded shared working memory. Two graph-aware selector variants (co-occurrence and citation graphs) constrain the reading order to capture inter-paper relationships explicitly, achieving state-of-the-art on the OARelatedWork benchmark.

Problem

Existing related work generation (RWG) methods feed only truncated paper portions (abstracts, introductions, retrieved spans) due to LLM context-window limits, causing hallucinations and shallow comprehension. Even LLMs with extended windows (128K–200K tokens) degrade when context approaches their limit. Additionally, prior work fails to explain inter-paper relationships coherently, producing isolated per-paper descriptions instead of integrated comparisons.

Method

The framework operates on full-text references \(R = \{R_1, \ldots, R_N\}\) and the citing paper \(R_0\), where each paper is segmented into sections \(R_i = \{s_{i,1}, \ldots, s_{i,L_i}\}\).

  • Selector: At each step \(t\), selects the next \((R_t, s_t)\) tuple based on all paper abstracts, current working memory \(M_{t-1}\), and reading history \(H_{t-1}\). Outputs <End> when sufficient. Five strategies are evaluated: Sequential (SR), Random (RR), Vanilla LLM, Graph-Co, Graph-Ci.
  • Reader: Processes the selected section, updates the shared working memory \(M_t\), subject to a hard token cap (4096 tokens) that forces the reader to discard irrelevant content at each step.
  • Writer: Generates the final RWS from terminal memory \(M_T\) and history \(H_T\), prompted with explicit instructions (avoid isolated descriptions, group similar works).

The graph-aware selector constrains navigation to one-hop subgraphs \(G_{t-1}\) of either: - A co-occurrence graph \(G_\text{co}\): edges connect papers co-cited in the same sentence of any reference's RWS. - A citation graph \(G_\text{ci}\): edges follow direct citation relationships among \(\{R_0\} \cup R\).

At each step the selector chooses to stay on the current paper or jump to an adjacent node, with access to the full graph \(G\) for global context.

Key Contributions

  • Multi-agent Selector–Reader–Writer pipeline that processes full-text references iteratively, avoiding single-pass long-context degradation.
  • Bounded shared working memory with forced reorganization, enabling unlimited reference depth within a fixed token budget.
  • Two graph-aware selector strategies that explicitly encode inter-paper structure (co-occurrence and citation) into the reading order.
  • Comprehensive evaluation combining graph-based structural metrics (avg. edges, avg. node degree, clustering coefficient) and LLM-based evaluation (coverage, logic, relevance) across three judge models.

Results

All numbers from OARelatedWork dataset; 10% subset used due to scale.

  • Graph-Ci consistently best: GPT-4o Graph-Ci reaches 3.87 Overall (LLM-based), vs. 3.75 for GPT-4o+GO (retrieval baseline) — +3.2% improvement.
  • Claude-3-Haiku Graph-Ci: 3.61 Overall vs. 3.52 (retrieval baseline) — +2.6%.
  • Llama3-8B Graph-Ci: 3.44 Overall vs. 3.29 (retrieval baseline) — +4.6%.
  • Feeding full text in a single pass to Claude-3-Haiku degrades overall LLM-based score by 9.3% vs. abstract-only input.
  • Graph-Ci outperforms Graph-Co across all three base models on both graph-based and LLM-based metrics.
  • Selector ranking across all models: SR < RR < Vanilla < Graph-Co < Graph-Ci.
  • Human evaluation win-rates of Graph-Ci vs. retrieval baseline: Llama3-8B 61.3%, Claude-3-Haiku 59.7%, GPT-4o 58.7%; largest margin on Logic dimension.

Limitations

  • Evaluated on only 10% of OARelatedWork due to computational cost; results may not fully generalize.
  • OARelatedWork is currently the only full-text RWG dataset; cross-dataset generalization is untested.
  • The 4096-token memory cap forces lossy compression; information discarded by the reader is unrecoverable.
  • Citation and co-occurrence graphs require access to reference papers' RWS sections, which may not always be available.
  • Weak base model (Llama3-8B Graph-Ci) still underperforms abstract-only Claude-3-Haiku, indicating the framework amplifies but cannot compensate for base model capability gaps.
  • Graph construction depends on known citation/co-occurrence structure; handling papers with sparse citation links is not addressed.

Relevance to Harnesses / Meta-Harnesses

This work is a direct instantiation of an agentic harness pattern: a controller (selector) dispatches read operations over a corpus, a processor (reader) updates shared mutable state (working memory), and a synthesizer (writer) consumes accumulated state to produce output — precisely the orchestrator–worker–aggregator topology common in meta-harnesses. The bounded working memory with forced eviction at each step is a concrete solution to the context-accumulation problem any long-running harness must address. The graph-aware selector demonstrates that imposing structured traversal constraints on an agent's action space — rather than leaving selection fully open-ended — substantially improves output quality, which is a design principle directly applicable to harnesses that schedule tool calls or sub-agent invocations over structured knowledge graphs.