Skip to content

From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes replacing absolute-score-based LLM peer review with a pairwise comparison mechanism: LLM agents compare manuscript pairs, and Bradley-Terry model aggregation produces a global quality ranking. At sufficient scale (~10^5 comparisons on ICLR 2024's 7158-paper pool), the system matches human acceptance quality by citation count but introduces measurable biases toward established institutions and mainstream topics.

Problem

Existing LLM-based peer review systems replicate the traditional workflow—independent per-paper scoring by LLM reviewers followed by meta-reviewer aggregation—which suffers from poorly calibrated absolute ratings (GPT-4o mini ratings cluster around 6–7, limiting discriminative power) and misses the opportunity to redesign evaluation rather than merely automate it.

Method

Each of M LLM agents (GPT-4o mini) is assigned a randomly sampled paper pair and produces a binary preference judgment. All comparison outcomes are collected as triplets (i, j, y_ij). The Bradley-Terry model assigns each paper a latent quality score β_i; the probability that paper i beats paper j is p_ij = e^{β_i} / (e^{β_i} + e^{β_j}). Scores are estimated via maximum likelihood on the observed comparisons, and papers are ranked by descending β. Acceptance decisions are made by thresholding the global ranking. The method needs fewer than 2% of all possible pairs to recover a high-quality ranking (supported by theoretical sample complexity bounds from Negahban et al.).

Key Contributions

  • First exploration of a pairwise-comparison paradigm for LLM peer review, as an alternative to score-based replication of traditional workflows.
  • Empirical demonstration of a citation-count scaling law: average citations of accepted papers increase steadily from 10^3 to ~10^5 agents, plateauing near 20.00 (vs. 19.36 for human, 11.41 for GPT rating system, 8.56 for random).
  • Consistency analysis showing 41.0% overlap with human ICLR 2024 accepts, on par with NeurIPS 2021 human-human committee agreement (48%).
  • Bias characterization: GPT ranking over-accepts applied/robotics papers (0.56 vs. 0.32) and under-accepts theoretical/learning-theory papers (0.12 vs. 0.50); shows reduced topic novelty (lower nearest-neighbor embedding distance among top-ranked papers) and higher institutional Gini coefficients (e.g., 37.2% of top selections from 10 institutions vs. 26.7% for humans at ICLR 2024).

Results

  • Citation quality: GPT ranking without area control achieves 20.00 mean citations for accepted papers vs. 19.36 (human), 11.41 (GPT rating system), 8.56 (random) on ICLR 2024.
  • Discriminative power: Statistically significant citation gaps between high-tier and low-tier GPT-ranked papers replicated across ICLR 2024, ICLR 2023, EMNLP 2023, CoRL 2023.
  • Human agreement: 41.0% (ICLR 2024), 42.2% (ICLR 2023) overlap with human accepts.
  • Topic novelty: GPT top-ranked papers have significantly smaller nearest-neighbor embedding distances than human-selected papers (p < 0.05 on ICLR 2023, ICLR 2024, EMNLP 2023).
  • Institutional inequality: Gini coefficients for GPT-selected papers consistently higher than human-selected across all five conferences studied.

Limitations

  • Citation count is an imperfect proxy for quality—affected by community size, visibility, and the Matthew effect; human-accepted papers also receive a presentation-visibility boost the GPT system does not.
  • The system exhibits systematic area bias (favoring applied over theoretical work), potentially due to LLM training-data distribution and weak mathematical reasoning.
  • Reduced topic novelty among GPT-ranked papers risks penalizing genuinely novel but unconventional research.
  • Pronounced institutional concentration could exacerbate existing prestige hierarchies in academic publishing.
  • Evaluated only with GPT-4o mini; generalizability to other LLMs is untested.
  • No direct human evaluation of decision quality; all empirical validation relies on proxy metrics.

Relevance to Agentic AI / LLM Agents

This work demonstrates a concrete multi-agent evaluation architecture where LLM agents function as distributed, stateless comparators whose individual judgments are aggregated into a coherent global decision via a statistical model—illustrating how agent-scale deployment (10^5+ calls) can unlock capabilities unavailable to single-agent systems. It is directly relevant to the broader question of how to design LLM-agent pipelines for high-stakes evaluation tasks beyond chat, showing both the scaling dynamics and the systematic biases that emerge when agents collectively make consequential decisions. The emergent biases (institutional concentration, topic homogenization) are a concrete case study in alignment challenges for agentic systems operating at scale without human oversight in the loop.