Skip to content

MissingBench-Verified: Probing Vision-Language Models' Inability to Detect Missing Object Parts

🕒 Published (v1): 2026-07-21 03:43 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; benchmark exposing VLM failure on missing-part detection; honest hallucination audit

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MissingBench-Verified is a 118-image benchmark probing whether frontier VLMs can detect essential missing object parts (e.g., an airplane without engines). All ten tested models fail, with correct detection rates below 75%, and no existing mitigation strategy—tool injection, image processing, extended reasoning, or fine-tuning—meaningfully closes the gap.

Problem

Prior hallucination benchmarks focus on weakly associated missing objects (a skiing scene without a human), which are plausible configurations VLMs can acknowledge. Objects missing essential parts (an airplane without engines, an iPhone without a home button) violate strong learned priors, creating a harder knowledge-vs-perception conflict that existing benchmarks undertest. Evaluation has also concentrated on small 7B–13B models rather than frontier production systems.

Method

The authors manually curate 118 real-world or internet images, remove a salient essential part using either Nano Banana 2 (AI image editing) or cleanup.pictures, and apply human verification to filter failed edits. Ten frontier models (5 proprietary, 5 open-source) are queried via OpenRouter under four conditions: 1. Direct query — plain prompt with CoT, structured JSON output (3-way: Clearly Visible / Hard to See / Not Visible). 2. Pre-answer tool injection — a simulated perfect object detector result (always "undetected, confidence < 0.1") is injected before the model answers. 3. Post-answer tool injection — same simulated result injected after an initial response, testing confirmation bias. 4. Thinking with images — the model controls cropping, brightness/contrast, and binarization tools before answering (tested on Gemini 3 Flash, Qwen 3.5 Flash, Qwen 3.5 Plus).

GPT-5.2 is additionally tested across six reasoning-effort levels (None → xHigh). OWLv2 and YOLOE are evaluated as traditional detection baselines using AP, AUROC, and threshold-adjusted accuracy.

Key Contributions

  • A human-verified benchmark specifically targeting essential missing parts, harder than prior hallucination datasets (frontier models score ~75–90% on RH-Bench vs. <75% on MissingBench-Verified).
  • Systematic demonstration that five mitigation strategies—direct prompting, simulated perfect detector injection (pre/post), image processing tools, and increased reasoning effort—all fail to achieve acceptable accuracy.
  • Qualitative failure analysis showing models rationalize contradictions via occlusion attribution, cropping artifacts, or tool dismissal rather than updating their visual judgment.
  • Comparison with traditional detectors (OWLv2, YOLOE), showing they fail differently (lack of knowledge/detection rather than hallucination) and may be more correctable.

Results

  • Direct query accuracy (Not Visible): all models below 75%; range 42.5% (NegAwareQwen-27B) to 75.4% (Gemini 3.1 Pro Preview); Claude Sonnet 4.6 lowest at 44.1%.
  • Pre-answer tool injection: best gains from Qwen 3.5 Flash and Plus (+16.9% each), Kimi K2.5 (+14.4%); most models remain below 75% NV.
  • Post-answer tool injection: Gemini 3.1 Pro reaches 82.2% NV (sole model above 75%); Qwen 3.5 Plus +24.5%; most others show marginal or noisy change.
  • Image processing tools: NV improvement ≤4.2% (Qwen 3.5 Flash); Qwen 3.5 Plus shows −0.9% (noise).
  • Reasoning effort (GPT-5.2): NV ranges from 57.6%–66.1% across None→xHigh; proportions z-test p=0.174 between most extreme groups—not significant.
  • RH-Bench comparison: Gemini 3 Flash 86.0%, Qwen 3.5 Flash 75.6%, Gemini 3.1 Pro 89.8% on RH-Bench vs. substantially lower on MissingBench-Verified, confirming harder difficulty.
  • OWLv2: AP=0.745, AUROC=0.710, max accuracy=0.657; suffers semantic confusion at lowered thresholds. YOLOE: AUROC=0.670, nearly misses all originals at default threshold (8% original detection rate).

Limitations

  • Dataset contains only 118 images; authors explicitly note the benchmark is existence-proof rather than a precise leaderboard.
  • No automated data pipeline; scaling requires future work.
  • Hallucination-specific training interventions not explored (no separate training set; training frontier models impractical).
  • Models evaluated in default configuration only; prompt engineering variations not systematically ablated.
  • NegAwareQwen fine-tuning experiment confounded by domain mismatch (trained on easy VSF cases) and model size.

Relevance to Vision-Language Models

This work isolates a structurally distinct and severe failure mode in frontier VLMs—parametric prior override in the face of implausible-but-real visual configurations—that scales with model capability (larger models still fail) and resists all post-hoc interventions tested. For VLM researchers, it establishes that the knowledge-vs-perception conflict is not a prompting or reasoning artifact but likely embedded in visual encoders or cross-modal alignment during pretraining. The finding that even a simulated perfect detector injected pre-answer cannot reliably correct responses directly motivates research into architectural mechanisms for grounding VLMs in visual evidence over learned priors, and raises concerns for any safety-critical deployment (inspection, monitoring, quality control) relying on VLMs for anomaly detection.