Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarks¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; multi-agent debate reduces LLM overconfidence in IR annotation labeling

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

DREAM is a multi-agent debate framework for automated relevance labeling in IR benchmarks, where two LLM agents initialized with opposing stances (relevant vs. irrelevant) debate iteratively, auto-labeling cases on agreement and escalating only persistent disagreements to humans. Applied to BEIR and RobustQA, it achieves 95.2% balanced accuracy with only 3.5% human involvement, uncovering 29,824 missing relevant chunks to form the BRIDGE benchmark.

Problem¶

IR evaluation benchmarks suffer from "holes"—unlabeled but truly relevant chunks—because human annotation is expensive and only covers a small corpus fraction. Existing LLM-based solutions either (a) fully automate labeling and propagate single-model overconfidence errors, or (b) use confidence-based AI-human hybrid escalation that is poorly calibrated: miscalibrated scores over-escalate easy cases and miss genuinely hard ones, requiring up to 50% human involvement to match DREAM's quality.

Method¶

DREAM structures relevance labeling as a three-stage protocol over query–chunk pairs $(q, c)$ with associated answer set $A(q)$:

Opposing Stance Initialization: Two LLM agents $m_1, m_2$ are assigned fixed opposing stances $s_1 = \text{relevant''}$ and $s_2 = \text{irrelevant''}$, forcing divergent evidence surfacing and preventing premature consensus.
Multi-Round Reciprocal Debate: In each round $j \leq R$, each agent critiques the opponent's prior reasoning, extracts evidence sentences from the chunk, and produces a new label $y_i^j$ with rationale $r_i^j$. The debate history $h^j = \{(r_1^{j-1}, y_1^{j-1}), (r_2^{j-1}, y_2^{j-1})\}$ is passed as context. If $y_1^j = y_2^j$ at any round, the agreed label is accepted: $$\text{DREAM}(q,c) = \begin{cases} y_1^j, & \exists\, j \leq R \text{ s.t. } y_1^j = y_2^j \\ \text{Human}(q, c, h^R), & \text{otherwise} \end{cases}$$ Default $R = 2$ (ablations show no gain beyond).
Agreement-Based Human Escalation: Persistent disagreement after $R$ rounds triggers escalation; the annotator receives the full debate history to contextually assist judgment—shown to raise inter-annotator agreement (Fleiss $\kappa$ from 0.50 to 0.62).

Implementation uses two Llama3.3-70B-Instruct instances (temperature 0.0). BRIDGE is built by applying DREAM to candidate chunks retrieved by 25 diverse retrievers over BEIR and RobustQA.

Key Contributions¶

DREAM framework: multi-agent adversarial debate for IR relevance labeling, replacing confidence calibration with inter-agent agreement as the escalation signal.
BRIDGE benchmark: re-labeled BEIR + RobustQA, uncovering 29,824 missing relevant chunks—428% of the original 6,976 gold chunks.
Bias analysis: demonstrates that holes in IR benchmarks distort retriever rankings and cause apparent retrieval–generation misalignment in RAG.
Cost efficiency: 200× cheaper and 3.5–7.0× faster than full non-expert (MTurk) human annotation.
AI-human synergy: debate history provided to escalated-case annotators yields measurable accuracy gains over annotation without history.

Results¶

DREAM: 95.2% balanced accuracy (bAcc), 3.5% escalation ratio — surpasses Human-Only (MTurk) at 93.8% bAcc with 100% human cost.
LLMJudge (single-agent, no escalation): 73.9% bAcc, 0% escalation — severely underperforms due to overconfidence on irrelevant cases.
LARA (confidence-based hybrid): requires 50% escalation ratio to approach 96.3% bAcc; at matched 3.5% escalation achieves only 82.1% bAcc.
Ablation — debate rounds: $R=1$ → 90.0% bAcc (LLM adjudicator); $R=2$ → 93.3%; $R=3$ → 93.2%; gains saturate at $R=2$.
Ablation — debate history for humans: with history, bAcc on escalated cases = 92.0% (Fleiss $\kappa = 0.62$); without, 87.3% ($\kappa = 0.50$).
Diversity ablations: adding more agents, heterogeneous LLM families, or higher temperature all degrade accuracy.
BRIDGE impact: retriever rankings shift meaningfully once holes are filled; retrieval–generation misalignment in RAG is partly explained by misestimated retrieval performance under incomplete labels.

Limitations¶

Only tested with Llama3.3-70B-Instruct; generalization to other model families or smaller LLMs is not demonstrated in the main results.
Maximum $R = 2$ chosen empirically; no theoretical bound on convergence behavior.
Increasing debate diversity (more agents, heterogeneous families, higher temperature) consistently hurts performance, limiting the design space for scaling.
BRIDGE covers only BEIR and RobustQA subsets; holes may persist in other benchmark domains not examined.
Escalated cases still require human annotators; the 3.5% fraction may not scale cheaply to extremely large corpora.
Spurious consensus (agents agreeing incorrectly) is not consistently reduced by more rounds, leaving a residual error floor in automated labels.

Relevance to Agentic AI / LLM Agents¶

DREAM is a concrete instantiation of structured multi-agent debate for a high-stakes annotation task, demonstrating that initializing agents with adversarial stances and using inter-agent agreement as a reliability signal outperforms single-agent confidence calibration—a design lesson broadly applicable to agentic pipelines that require self-assessment or quality filtering. The agreement-based escalation mechanism offers a calibration-free alternative to threshold tuning for routing decisions in human-in-the-loop agentic systems. The finding that debate history improves downstream human judgment (rather than just replacing it) is a novel data point for AI-human collaboration architectures. For researchers building agentic IR or RAG pipelines, BRIDGE also provides a less-biased retrieval evaluation benchmark that can expose evaluation artifacts in retriever-agent designs.