Skip to content

LiRA: A Multi-Agent Framework for Reliable and Readable Literature Review Generation

🕒 Published (v1): 2025-10-01 12:14 UTC · Source: Arxiv · Venue: AAAI · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LiRA is a multi-agent LLM workflow that decomposes systematic literature review writing into four specialized agents (outline drafter, subsection writer, editor, reviewer) with iterative feedback loops. It outperforms AutoSurvey and MASS-Survey on citation quality and structural coherence across two datasets while producing ~2.3× shorter output than the strongest baseline. No domain-specific fine-tuning is required.

Problem

Automated SLR tools focus on retrieval and screening; the writing phase remains under-addressed, particularly regarding hallucination/citation accuracy and readability. Existing agentic writing frameworks (AutoSurvey, MASS-Survey) inflate metric scores via verbosity and lack explicit factual grounding mechanisms.

Method

LiRA decomposes review generation into a sequential pipeline of four LLM agents, each with private memory:

  1. Outline Drafter: reads up to 50 reference abstracts/full texts, generates candidate outlines (~8 sections, ~4 subsections), combined into a single structured draft.
  2. Subsection Writer: writes each subsection in parallel (~1,000 words each); retrieves relevant references per subsection using FAISS semantic search, capped at 25% of total pool (min 3, max 150). Title/abstract/conclusion are written after body sections to avoid premature commitment.
  3. Editor: refines fluency, transitions, and style without altering factual content; uses a continuation mechanism when output exceeds 16,384 tokens to effectively double generation capacity.
  4. Reviewer: evaluates each component against adapted SLR quality criteria (completeness, clarity, contribution), triggers regeneration for up to 3 rounds before fallback.

Citation grounding: agents cite sources by full article title rather than numbered placeholders; post-processing maps these to numbered references and redacts hallucinated titles. All agents use Zero-Shot Chain-of-Thought prompting (except researcher and editor). Inter-agent communication follows MetaGPT-style structured document passing through shared message pools. Implemented in LangGraph.

Key Contributions

  • First agentic LLM pipeline for SLR writing that explicitly models the human writing process (analyze → outline → draft → edit → review).
  • Integration of formally defined SLR guidelines into agent prompts, bridging methodology literature and NLP system design.
  • Citation-grounding via full-title anchoring to mitigate hallucination without domain-specific training.
  • State-of-the-art baselines established on SciReviewGen and a proprietary ScienceDirect dataset (125 reviews each, cross-domain generalization to 23 subject areas).
  • Robustness analysis: cross-model reviewer (gemma3:4b) and retrieval-augmented settings evaluated.

Results

Citation Quality (CQF1) — primary reliability metric: - LiRA: 0.76 (SciReviewGen), 0.73 (ScienceDirect) - AutoSurvey: ≤0.63 on both; MASS-Survey: 0.13/0.33; Direct Prompting: 0.14/0.06

Writing Quality (Prometheus 2 LLM evaluator, 1–5 scale): - LiRA average: 4.13 (SciReviewGen), 3.88 (ScienceDirect) vs. AutoSurvey 3.78/3.53, MASS 3.83/3.39, DP 3.72/3.43 - LiRA leads on Structure (3.38 vs. AS 2.30 on SciReviewGen); AS marginally better on Coverage due to verbosity (~50k tokens vs. LiRA's ~22k)

Similarity to human reviews (ROUGE-L): - LiRA: 0.13 (both datasets); AutoSurvey: 0.09/0.13; MASS: 0.09/0.04

SME evaluation: experts preferred LiRA over human-written reviews on SciReviewGen for coverage/balance; on ScienceDirect, LiRA scored highest on structure.

Cross-model reviewer (gemma3:4b): negligible metric change, confirming pipeline robustness to reviewer model variation.

Retrieval setting: only 2 of 7 metrics significantly degraded vs. gold-reference baseline.

Limitations

  • Evaluated exclusively with gpt-4o-mini; performance with larger/stronger models untested in main experiments.
  • Recall-based similarity metrics (hsr, her, aer) conflate verbosity with coverage; the paper's length-normalization argument is qualitative, not metric-corrected.
  • Up to 50-reference cap in outline drafting may exclude relevant papers from very large corpora.
  • Self-bias amplification in the reviewer-writer loop (same underlying model) is acknowledged but not fully quantified by existing metrics.
  • Evaluation on proprietary ScienceDirect data is not reproducible externally.
  • No latency/cost analysis reported; parallelization details are high-level.

Relevance to Harnesses / Meta-Harnesses

LiRA is a direct instance of a multi-agent harness that pipelines specialized sub-agents (planner, writer, editor, reviewer) with iterative feedback loops and shared message pools — an architecture pattern central to harness/meta-harness design. The reviewer agent's loop-with-fallback (up to 3 refinement rounds) exemplifies controlled self-correction within a harness, analogous to meta-harness retry and validation mechanisms. The integration of MetaGPT-style structured inter-agent communication and FAISS-driven dynamic context routing are reusable harness primitives. For researchers building harnesses over LLM agents, LiRA demonstrates how formally defined external guidelines (SLR criteria) can be operationalized as quality gates within an automated pipeline, a transferable pattern for any domain requiring structured long-form synthesis.