Skip to content

Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models

🕒 Published (v1): 2026-07-09 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

Blind-Spots-Bench; systematic evaluation of multimodal model failure modes humans find trivial; benchmark for VLM robustness

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Blind-Spots-Bench is a 235-sample benchmark of tasks that are trivially easy for humans yet consistently fail frontier AI systems, sourced from graduate students who elicited failures from live models. It covers text-only, multimodal (image+text → text), and image-generation formats with a structured taxonomy. Closed-source models outperform open-weight counterparts by ~10% even when AAII scores are matched, and no single model dominates across all task types.

Problem

Existing benchmarks are near-saturated by frontier models, masking persistent failure modes—e.g., counting objects in images, generating a string of exact length, spatial composition in image generation—that any human solves effortlessly. There is no unified benchmark that simultaneously covers text, vision-language, and generative image tasks through a fine-grained diagnostic taxonomy targeting these natural failure modes rather than adversarial perturbations.

Method

Questions were crowd-sourced from students in a graduate AI course (October 2025) who each submitted five problems on which frontier chatbots failed. After deduplication, difficulty thresholding, and three-stage quality review, 235 samples were retained. Each sample is annotated with: (1) a structured reference solution specifying expected answer, correctness conditions, and common failure modes; (2) a question-format label (text-only, multi-to-text, image-gen); and (3) a fine-grained task label from a three-level taxonomy (Object-centric → perceptual counting, spatial reasoning, attribute/pattern recognition, generative counting; Abstract reasoning → arithmetic, logical, geometric/graph; Language and knowledge → character-level manipulation, etc.; 12 subcategories total).

Evaluation uses a two-stage pipeline built on Inspect AI: a solver model answers without chain-of-thought or in-context examples; a grader (Gemini-3-Flash with code execution enabled) produces binary correctness judgments. Human–grader agreement is 96.6% on textual tasks and 90.9% on image-generation tasks; no evidence of pro-Google grader bias was found. Metrics reported are mean@4 (average accuracy over 4 independent runs) and pass@4. 38 models are evaluated: 32 LLMs/VLMs and 6 image-generation models, all with thinking mode at medium effort.

Key Contributions

  • Blind-Spots-Bench dataset: 235 curated open-ended problems spanning text-only (46.2%), image-gen (35.6%), and multi-to-text (18.2%) formats, publicly available on HuggingFace.
  • Task taxonomy: 3 high-level categories × 12 fine-grained subcategories tailored to naturally occurring model failure modes, enabling per-subtask breakdown.
  • End-to-end evaluation pipeline: automated grader with code execution, human-validated agreement, bias-tested; covers 38 models including the full GPT-5.x and Gemini-3.x families.
  • Systematic comparison: largest publicly reported evaluation of the GPT-5.x/Gemini-3.x/GLM-5.x/Qwen3.5 generation on a single human-trivial benchmark.

Results

  • Best overall text-only: GPT-5.5 ≈84.0% mean@4; Gemini-3.1-Pro ≈83.3%.
  • Best overall multi-to-text (VLM): Gemini-3.1-Pro ≈66.9%; GPT-5.5 ≈58.7% (strong text performance does not transfer to visual questions).
  • Best open-weight VLM (text-only): GLM-5.2 ≈73.8%; Qwen3.5-397B ≈71.1%.
  • Open-weight VLM gap: top open-weight lags top closed-source by ~10 pp on text-only, larger on multi-to-text, even at comparable AAII scores.
  • Image generation: Gemini-3-Pro-Image 54.8% vs. GPT-Image-2 51.2% at ~4× lower cost for GPT.
  • Perceptual counting: no model exceeds 60% accuracy, making it a universal blind spot.
  • Cost-performance: a 10-pp gain (60% → 70%) on text-only tasks requires ~10× higher inference cost; DeepSeek-V4, Qwen3.5, GLM-5.2 are Pareto-efficient among open-weight models.
  • Tool use (code execution): mixed—Gemini-3.1-Flash-Lite +9.03 pp, GPT-5.4 −5.32 pp; effect is model-dependent.
  • Scaling within families: larger model size does not consistently improve performance.

Limitations

  • Dataset is small (235 samples); per-subtask analysis has high variance, particularly for multi-to-text (\(n \approx 43\) examples).
  • Image-generation evaluation runs only \(k=1\) (no pass@k reliability estimate) due to cost.
  • No open-weight image-generation models included due to technical constraints.
  • Questions originate from a single institution's graduate course, potentially reflecting a non-diverse cultural/linguistic sample.
  • All models evaluated with thinking enabled at medium effort; results may not generalize to zero-shot or zero-thinking settings.
  • Grader is Gemini-3-Flash; while bias was checked, grader errors contribute to the ≈3–9% disagreement with human labels.

Relevance to Vision-Language Models

Blind-Spots-Bench directly stress-tests VLMs on the subtask categories they are known to struggle with—perceptual counting, spatial reasoning, and attribute binding in image inputs—providing a more discriminating signal than near-saturated benchmarks like MMMU or MM-Vet. The finding that GPT-5.5 achieves ~84% on text but only ~59% on multi-to-text problems quantifies the visual grounding gap even in the most capable current VLMs, and the per-subtask leaderboard (Fig. 1 right) gives practitioners a concrete map of where each model family fails. For researchers developing or evaluating VLMs, this benchmark offers a reproducible, low-cost diagnostic to complement aggregate scores.