Skip to content

Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Models

🕒 Published (v1): 2026-07-10 17:53 UTC · Source: Arxiv · link

Why this paper was selected

Decade-long longitudinal analysis of VLM visual-cognitive errors; rare rigorous error taxonomy work

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper benchmarks nine vision-language models (2017–2025) on scene description accuracy and a novel taxonomy of five visual-cognitive error types, using a new Complex Social Behavior (CSB) dataset of 100 movie-frame images depicting complex social interactions. Pre-MLLMs fail dramatically on CSB while matching human bottom-tier performance on MS-COCO; MLLMs close the gap and reach top-human accuracy on both datasets. The one persistent failure mode across all models is spatial dependence — models rely on different image regions than humans do.

Problem

Standard VLM benchmarks (e.g., MS-COCO) use simple, object-centric scenes that do not include complex human social interactions or behaviors, masking systematic visual reasoning deficits. Additionally, prior work rarely categorizes which types of errors have been eliminated versus which persist, making it hard to understand residual model weaknesses.

Method

Dataset. The authors construct the CSB dataset: 100 images sampled as frames from commercially released films depicting complex social behaviors, collected by 10 research assistants. A 100-image MS-COCO subset serves as a control.

Gold standard. For each image, 20 descriptions are collected via Amazon Mechanical Turk. A separate group of 50 workers rank the 20 descriptions by semantic fidelity; the most-voted description becomes the gold standard. Top-4 and bottom-4 of the remaining 19 are kept for human-range bounds.

Evaluation metric. Gemini-SI and GPT-SI: cosine similarity between 3,072-dimensional sentence embeddings from Gemini and GPT, respectively. These LLM-based similarity metrics are validated against human pairwise ratings and shown to outperform BERTScore correlation with human judgments.

Models. Four pre-MLLMs (UDC, M2M, OSCAR, OFA — all ResNet-101-based, 2017–2022) and five MLLMs (GPT-4, Gemini, GPT-4o1, GPT-5.1, Gemini3). A standard text prompt ("Make your best guess of what might be happening in this scene in one sentence…") is used for the six models requiring text input.

Error taxonomy. Five error types are measured: - Detection: relevant object absent from description and lacks bounding box. - Recognition: object detected but misclassified (or wrong object hallucinated via misclassification). - Scene understanding: object correctly detected and recognized but omitted from description. - Hallucination: object described that is not present in the scene. - Spatial dependence (SDE): model and humans rely on different image regions. Measured via a masking paradigm — three occlusion masks covering ~1/3 of each image are applied; model–human correlation across masked-region similarity scores is compared to inter-human correlation. SDE is flagged when the model–human correlation falls \(>2\sigma\) below the inter-human baseline.

Bootstrapping (10,000 resamples) and FDR correction (Benjamini–Hochberg) are used for inference.

Key Contributions

  • Introduces the CSB dataset: 100 complex-social-behavior images from films, with 20 ranked human descriptions per image and a rigorously derived gold standard.
  • Proposes Gemini-SI/GPT-SI, LLM-embedding cosine-similarity metrics that correlate better with human raters than BERTScore.
  • Provides a decade-long longitudinal analysis (2017–2025) of nine models, including early models no longer available on current platforms.
  • Introduces a five-category visual-cognitive error taxonomy (detection, recognition, scene understanding, hallucination, spatial dependence) applied consistently across pre-MLLMs and MLLMs.
  • Demonstrates via an in-house, never-posted dataset that MLLM performance is not explained by training-set memorization of movie frames.

Results

  • Pre-MLLMs score significantly below bottom-ranked human descriptions on CSB (\(p < 0.0001\), bootstrap); on MS-COCO they match bottom-ranked humans but still fall short of top-ranked.
  • All five MLLMs achieve accuracy comparable to top-ranked human descriptions on both CSB and MS-COCO datasets (Gemini-SI and GPT-SI metrics).
  • MLLMs eliminate the pre-MLLM accuracy gap between MS-COCO and CSB; pre-MLLMs show a large gap (\(p < 0.0001\)).
  • Pre-MLLMs make significantly more detection, recognition, and scene understanding errors on CSB than on MS-COCO (\(p < 0.001\)–\(0.01\)); MLLMs commit few such errors on either dataset.
  • Detection, recognition, and hallucination errors exert the highest impact on scene description accuracy across the decade.
  • Spatial dependence is the only error category that persists in MLLMs — models occasionally attend to different image regions than humans when forming descriptions.
  • Results replicate on the in-house CSB images (unseen at training time): MLLM performance remains near human level; OFA (pre-MLLM) remains significantly below (\(p < 0.0001\)).

Limitations

  • The CSB dataset is small (100 images), necessitating bootstrap inference; generalization to the full distribution of complex social scenes is uncertain.
  • Movie-frame images may carry domain-specific biases (camera angles, lighting, acting conventions) not representative of naturalistic social scenes.
  • MLLM internal architectures are proprietary, so error attribution relies on probing via targeted follow-up questions rather than mechanistic inspection.
  • Only scene description is evaluated; other VL tasks (VQA, referring expression, grounding) are not covered.
  • The masking paradigm for SDE uses only three mask positions per image, which is a coarse spatial resolution for measuring region importance.

Relevance to Vision-Language Models

This paper provides the most systematic longitudinal ablation of VLM error modes to date, revealing that the jump from pre-MLLM to MLLM architectures corresponds to near-elimination of detection, recognition, and hallucination errors on complex scenes — a direct signal for researchers studying scaling and architecture transitions. The finding that spatial dependence remains is actionable for VLM alignment work: models and humans attend to different scene regions even when producing semantically similar descriptions, suggesting a persisting gap in bottom-up saliency or attention mechanisms. The CSB dataset and Gemini-SI/GPT-SI metric both serve as useful additions to the VLM evaluation toolkit, particularly for benchmarking social and relational scene understanding beyond object-centric tasks. The in-house memorization control strengthens the credibility of MLLM accuracy claims that have often been questioned in the benchmarking literature.