Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verification¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; agreement bias mitigation via self-grounded verification in MLLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

MLLMs used as verifiers for agentic tasks exhibit a systematic "agreement bias"—over-validating flawed agent trajectories even when they possess human-aligned priors about correct behavior. The paper introduces Self-Grounded Verification (SGV), a two-step inference method that first elicits task priors unconditionally, then evaluates trajectories conditioned on those priors. SGV improves failure detection by up to 25 pp and downstream task completion by up to 33% relative across web, desktop, and robotic environments.

Problem¶

MLLM verifiers for open-ended agentic tasks (web navigation, desktop, robotics) suffer from a pervasive failure mode: they systematically over-validate incorrect agent behavior (high false-positive rate, true-negative rates as low as 50%). This "agreement bias" persists across 13+ model families, 28+ prompt/scoring templates, chain-of-thought, set-of-marks, majority voting, and test-time scaling with reasoning models. Because many pipelines—filtered behavior cloning, Reflexion-style self-improvement, online supervision, benchmark evaluation—rely on MLLM rewards, this bias cascades into degraded agent performance. The failure is paradoxical: models demonstrably hold correct priors about desired task behavior but fail to apply them when conditioned on the trajectory being evaluated.

Method¶

Self-Grounded Verification (SGV) modulates MLLM (un)conditional generation across two decoupled steps:

Step 1 – Prior generation. The MLLM is prompted with only the task $q$ and initial context states $s_{0:t}$ (no trajectory), eliciting a completion $\hat{k}_q$ about what successful task execution looks like: $$\hat{k}_q = g\!\left(\prod_{i=1}^n P(y_i \mid y_{<i},\, s_{0:t},\, C,\, q)\right)$$ This unconditioned generation allows the model to freely explore its knowledge distribution, producing an impartial reference prior.

Step 2 – Grounded evaluation. The trajectory $\tau_t$ is then evaluated conditioned on $\hat{k}_q$: $$r_{\text{SGV}}(\tau_t, C, q) = h\!\left(\prod_{i=1}^n P(y_i \mid y_{<i},\, q,\, \tau_t,\, C,\, \hat{k}_q)\right)$$ The self-generated prior acts as an anchor, shifting the conditional distribution toward more balanced, failure-sensitive judgments. No retraining or external supervision is required; SGV integrates directly into existing MLLM verifier pipelines.

Verifier performance is measured via bias $= \frac{1}{n}\sum_i \mathbb{E}[d_i]$, distance skewness $d_{\text{Skew}}$, true positive rate (TPR), true negative rate (TNR), and downstream task completion rate (SR).

Key Contributions¶

Agreement bias characterization: Documented across 13+ MLLMs/LRMs and 28+ evaluation templates on VWA (910 tasks), OSWorld (369 tasks), and robomimic; shows TNR as low as 50% and strongly positive bias/skewness even for models with >90% recall.
Downstream impact analysis: Shows that agreement bias concretely harms filtered behavior cloning, Reflexion-style self-improvement, and online supervision.
SGV: Training-free two-step inference method improving TNR by up to 25 pp and accuracy by up to 14 pp across all tested models and environments.
New SOTA on VisualWebArena: SGV-based online supervision surpasses previous best by 20 pp using only native web actions and lower token overhead.
Updated VisualWebArena: Corrected oracle verifiers (near-perfect agreement with human labels), high-fidelity environment parallelism, >10× runtime speedup, and VisualWebArena-Lite (1/3-scale subset with comparable fidelity).

Results¶

Offline trajectory evaluation (VWA + OSWorld): Baseline TNR across models: 34–78%; SGV improves TNR by up to 25 pp and accuracy by up to 14 pp for all 14 tested models.
Self-improvement (Reflexion on VWA): SGV yields up to +10 pp (24% relative) task completion improvement.
Online supervision (VWA – ReAct agent): +9 pp (20% relative) task completion.
Online supervision (OSWorld – UI-TARS-1.5 GUI specialist): +5 pp (22% relative).
Online supervision (robomimic tool-hang – diffusion policy): +8 pp (33% relative).
VisualWebArena SOTA: New best by 20 pp over previous best.
AgentRewardBench: Baseline MLLM verifier achieves SOTA; SGV further improves, surpassing the original VWA oracles.

Limitations¶

Experiments are confined to three environment families (VWA, OSWorld, robomimic); generalization to other interactive domains (mobile, embodied navigation) is not demonstrated.
Oracle verifiers used as ground-truth proxies are imperfect; VWA oracles required manual correction, and residual errors may persist.
SGV adds inference overhead (two sequential MLLM calls per verification); latency impact is not quantified in the provided text.
Agreement bias is identified as a phenomenon but its root causes (pretraining knowledge encoding, RLHF sycophancy) are attributed to prior work—no mechanistic proof or mitigation at training time is proposed.
Results on reasoning models (marked "(T)") are included but the full text of the paper is truncated, so some fine-grained ablation details are not available here.

Relevance to Vision-Language Models¶

This work directly characterizes a critical failure mode of VLMs when deployed as reward signals in agentic loops—a use case of rapidly growing importance as VLMs are integrated into RLHF, behavior cloning, and online supervision pipelines for GUI and robotic agents. The finding that agreement bias is resilient to chain-of-thought and test-time scaling challenges assumptions that stronger reasoning automatically yields better multimodal evaluation. SGV is a lightweight, plug-in solution for any VLM-as-verifier pipeline, making it immediately applicable to the broader community building on MLLMs as judges or reward models. The updated VisualWebArena benchmark also provides a higher-quality evaluation resource for future VLM agent research.