Skip to content

Precise but Uncoupled: Reviewer Precision Does Not Guarantee Critique Uptake in Multi-Agent Math Reasoning

🕒 Published (v1): 2026-07-16 18:38 UTC · Source: Arxiv · link

Why this paper was selected

4k+ example study proving reviewer precision decoupled from critique uptake — breaks a core MAS design assumption

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

In hierarchical Planner–Executor–Reviewer (PER) multi-agent pipelines for math reasoning, a high-precision reviewer does not guarantee that its critique actually changes the solver's next candidate. Broadcast peer deliberation achieves higher final accuracy (89.2% vs. 85.2%) than PER on 4,181 Omni-MATH problems despite PER's reviewer having superior precision (0.861 vs. 0.644), because PER's critique uptake rate is dramatically lower (33.6% vs. 93.5%). The paper introduces a measurement framework separating detection, uptake, and repair to make this "reviewer–solver decoupling" failure mode visible.

Problem

Hierarchical multi-agent systems that route critique through a dedicated reviewer role assume that reviewer quality translates into solver action ("role-specialization hypothesis"). Existing evaluations report only final pass rate or reviewer precision, masking a distinct failure mode: a reviewer can correctly flag an error yet the downstream solver can ignore the critique entirely. This decoupling is invisible under outcome-only metrics.

Method

The study is a test-time protocol comparison on Omni-MATH 2 (4,181 competition-level problems across 10 difficulty tiers), holding the model family fixed at gpt-oss-120b (temperature 0) and varying only the interaction protocol:

  1. Baseline LLM — one-shot, no retries.
  2. Single-Agent Iterative — repeated outer-loop attempts with failure/candidate memory, no reviewer.
  3. PER — Planner decomposes → Executor produces candidate → Reviewer decides submission or revision; reviewer feedback is routed as a separable advice field.
  4. Broadcast — three peers with confidence-polling for speech allocation; unanimous approval required before submission; critique enters shared candidate state.

Three within-PER variants probe the reviewer-to-solver interface while holding model, reviewer, and budget fixed: - ACK-required: mandatory verbal acknowledgment of critique (routed advice unchanged). - EMB: reviewer guidance placed directly in the solver's working context. - EMB + deeper reflection (appendix).

Metrics chain detection → uptake → repair: - \(\text{NeglectRate}\): fraction of wrong-initial-candidate cases where available critique is ignored. - \(\text{CouplingRate}\): fraction of evaluator-verified useful revise episodes where the protocol's next candidate actually changes. - \(\text{ReviewerGuidedRepairRate (RGR)}\): fraction of evaluator-verified useful critiques that lead to a correct repair.

Evaluator robustness is checked by replaying answers under two additional open evaluators (pairwise disagreement 3.38%–5.76%; \(\kappa = 0.850\)–\(0.915\)).

Key Contributions

  • Finding: PER's higher reviewer precision (0.861 vs. 0.644) does not translate to higher repair; its CouplingRate (0.336) and RGR (0.051) are far below broadcast's (0.934 and 0.286), confirming that reviewer quality and critique uptake are empirically separable.
  • Interventions: ACK-required (a compliance-style control) worsens final accuracy (85.2% → 82.5%) and CouplingRate (0.336 → 0.175); EMB partially recovers the gap (CouplingRate 0.227, FinalPassRate 86.3%) by placing reviewer guidance directly in solver context.
  • Measurement framework: Introduces the detection → uptake → repair decomposition with NeglectRate, CouplingRate, and ReviewerGuidedRepairRate as coupling-sensitive diagnostics applicable to any reviewer-to-solver interface.
  • Deployment view: Shifts evaluation away from reviewer quality alone toward whether the protocol acts on critique; includes verifier-cost analysis (PER is verifier-heavier at 2.19 vs. 1.35 avg. evaluator calls/problem despite lower token cost).

Results

  • Overall accuracy: Broadcast 89.2% > PER 85.2% > Single-Agent Iterative 78.8% > Baseline LLM 56.8%.
  • Collaboration gains by tier: Near zero on tiers 1–2; open sharply from tier 4 onward; on tiers 6–9, PER gains ~10–16 pp over Single-Agent, broadcast gains ~14–20 pp.
  • CouplingRate: PER 0.336 vs. broadcast 0.935 on evaluator-verified useful critique.
  • ReviewerGuidedRepairRate: PER 0.051 vs. broadcast 0.286 (5.6× gap).
  • NeglectRate: PER 48.8% vs. broadcast 26.2% of wrong-initial-candidate cases.
  • Post-failure recovery (conditional on first failure): broadcast 37.2% ≈ Single-Agent 37.0% > PER 32.4%.
  • ACK-required: Worsens NeglectRate from 0.488 → 0.792; CouplingRate 0.336 → 0.175; FinalPassRate 85.2% → 82.5%.
  • EMB: NeglectRate 0.698; CouplingRate 0.227; FinalPassRate 86.3% — partial recovery, gap to broadcast remains.
  • Resistance to false alarms: PER 92% vs. broadcast 71% (correct answers unchanged when reviewer falsely flags).
  • Cross-family replication (Gemma 3 actors, appendix): Overall ranking shifts but precision–uptake separation persists.

Limitations

  • Not a single-variable ablation: PER vs. broadcast differs in shared state, approval rules, communication topology, and discussion structure simultaneously; the gap cannot be attributed to one factor.
  • Within-PER probes are directional, not causal: ACK and EMB interventions change the interface but also other properties; they support the uptake story without isolating it.
  • Single model family in main experiments: gpt-oss-120b throughout; Gemma 3 cross-family replication is appendix-only and incomplete.
  • Benchmark scope: Omni-MATH is olympiad math; generalization to other domains (code, science, open-ended reasoning) is not established.
  • CouplingRate is protocol-level, not semantic: It measures answer-transition, not whether the model internally processed the critique; a candidate can change for reasons unrelated to the reviewer signal.
  • No training-time interventions: Entirely test-time; implications for RLHF- or fine-tuned reviewer–solver systems are not addressed.

Relevance to Agentic AI / LLM Agents

This paper directly challenges a widespread design assumption in LLM agent pipelines — that role specialization (dedicated reviewer) improves reasoning outcomes — by showing that information routing architecture matters as much as component quality: a higher-precision reviewer in a hierarchical pipeline can be outperformed by a lower-precision shared-state design because critique must actually bind to solver action to matter. The coupling-sensitive metric framework (NeglectRate, CouplingRate, RGR) is a practical diagnostic for any agent system with a critic/reviewer role, and the ACK-required failure demonstrates that surface compliance signals do not proxy for actual uptake. For researchers building or evaluating multi-agent reasoning systems, this work shifts the design question from "how good is the reviewer?" to "does the protocol architecture guarantee that critique reaches and changes the solver's next output?"