Automatic Paper Reviewing with Heterogeneous Graph Reasoning over LLM-Simulated Reviewer-Author Debates

🕒 Published (v1): 2025-11-11 14:46 UTC · Source: Arxiv · Venue: AAAI · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ReViewGraph frames automatic paper accept/reject prediction as heterogeneous graph reasoning over LLM-simulated multi-round reviewer-author debates. Three reviewer agents, one author agent, and a senior coordinator generate structured debates; typed opinion relations are extracted and encoded as a heterogeneous graph, then processed by a Heterogeneous Graph Transformer. The system outperforms all baselines by 15.73% relative Macro F1 on ICLR 2023–2025 datasets without any LLM fine-tuning.

Problem

Existing LLM-based paper reviewing methods either (a) use direct prompting, producing shallow, positively-biased, prompt-sensitive assessments, or (b) fine-tune open-source LLMs on scarce review data, yielding single-perspective outputs that miss the argumentative negotiation dynamics of real peer review. Both categories hallucinate and fail to model the discourse-level interactions between reviewers and authors that actually drive acceptance decisions.

Method

Multi-agent debate simulation: Three regular reviewer agents, one author agent, and one senior reviewer meta-coordinator execute a three-stage protocol — (1) initial review, (2) author rebuttal, (3) reviewer re-evaluation — using a multimodal LLM with full paper content.

Heterogeneous debate graph construction: The simulated dialogue is parsed via in-context prompting to extract opinion triplets and classify them into four evaluation dimensions (Methodological Novelty, Experimental Completeness, Motivation Clarity, Writing Fluency). Four node types are defined: Title, EvaluationDimension, ReviewerOpinion, AuthorOpinion. Four typed edge categories encode paper-dimension links, reviewer-dimension assignments, inter-reviewer relations (agree/disagree/complement/progressive/independent), and reviewer-author interaction relations (accept/reject/clarify/compromise/extend/neutral).

Graph reasoning: A Heterogeneous Graph Transformer (HGT) with relation-specific projection matrices performs heterogeneous mutual attention and message passing over the graph. Per-node-type mean pooling followed by concatenation feeds a two-layer MLP for accept/reject classification.

Key Contributions

• ReViewGraph framework: end-to-end pipeline from LLM debate simulation to heterogeneous graph construction to GNN-based decision prediction, requiring no LLM parameter updates.
• Semantically-typed heterogeneous debate graph with 4 node types and 4 meta-relation categories capturing fine-grained argumentative dynamics.
• Three new benchmark datasets from ICLR 2023/2024/2025 OpenReview with train/val/test splits and subcategory breakdowns (Oral/Poster/Spotlight).
• 15.73% average relative improvement over second-best baseline (CycleReviewer-70B) on Macro F1.

Results

• ICLR 2025: Acc 71.65, P 72.04, R 72.01, F1 71.65 vs. CycleReviewer-70B (Acc 63.05, F1 59.26) and DeepReview-14B (Acc 63.77, F1 54.34).
• ICLR 2024: F1 65.54 vs. DeepReview-14B F1 61.13 (second best).
• ICLR 2023: F1 69.89 vs. DeepReview-14B F1 58.69 (second best).
• Statistical significance confirmed vs. CycleReviewer-70B: accuracy p=0.0192, F1 p=0.0067.
• Average relative improvement of 15.73% across all three datasets over second-best baselines.
• Ablation: removing Title nodes causes the largest single drop; removing heterogeneity (collapsing to homogeneous graph) consistently degrades all metrics; removing either inter-reviewer (IRR) or reviewer-author (RAR) edges both reduce performance.

Limitations

• Evaluation is binary accept/reject classification only; the method is not assessed on review text generation quality.
• All three datasets are from ICLR only; generalization to other venues (NeurIPS, ACL, domain-specific conferences) is untested.
• The multi-agent debate simulation adds substantial LLM inference cost (three reviewer agents + author + senior reviewer, multi-round) compared to direct prompting baselines; no latency/cost analysis is provided.
• Opinion relation extraction and dimension classification rely on in-context prompting, inheriting prompt sensitivity and potential extraction errors that could corrupt the graph.
• Graph construction quality is not directly evaluated; errors in triplet extraction propagate silently to the GNN stage.

Relevance to Agentic AI / LLM Agents

ReViewGraph is a concrete multi-agent collaboration system where specialized LLM agents (reviewers, author, senior coordinator) execute distinct roles across structured interaction rounds — a direct instance of role-based multi-agent frameworks studied in agentic AI. The key architectural insight is that agent interaction traces can be lifted into structured symbolic representations (typed graphs) and then reasoned over with a separate GNN, rather than feeding raw dialogues to another LLM, which directly addresses hallucination and reasoning-depth problems common in LLM-only agentic pipelines. For researchers tracking agent coordination and debate-as-reasoning patterns, this work demonstrates that explicit relation extraction from agent dialogues yields measurably better downstream decisions than implicit dialogue modeling. It also connects to the broader question of how agentic systems should represent and propagate disagreement across agents — a key challenge for multi-agent consensus and alignment.