More Than Meets the Eye: Measuring the Semiotic Gap in Vision-Language Models via Semantic Anchorage¶

🕒 Published (v1): 2026-04-19 10:00 UTC · Source: Arxiv · Venue: ACL 2026 · link

Why this paper was selected

High visual fidelity hinders abstract VLM reasoning; semiotic gap study

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

VLMs consistently prefer literal visual interpretations over idiomatic ones—a "Literal Superiority Bias"—and this bias is driven in part by high visual fidelity acting as cognitive interference. The authors introduce DIVA, a schematic/iconographic benchmark derived from 100 English noun compounds, and a unified architecture-agnostic metric (Semantic Alignment Gap $\Delta$) to quantify and directionally characterize this bias across discriminative, open-generative, and proprietary VLMs.

Problem¶

Standard VL benchmarks use photorealistic imagery, which confounds semantic evaluation with surface-level texture and lighting cues. When VLMs encounter idiomatic noun compounds (e.g., "Eye Candy"), their object-detection circuits override symbolic reasoning, causing them to privilege literal constituent depictions over intended abstract meaning. Prior work (ADMIRE, T2I-ReasonBench) targets generation quality rather than quantifying the literal-vs-idiomatic alignment gap rigorously and across architectural families.

Method¶

DIVA benchmark construction: For each of 100 English idiomatic noun compounds (200 test instances, 1,000 total iconographic images), high-fidelity source images from the ADMIRE dataset are transformed into schematic iconographic renderings via a two-stage pipeline: (1) automated generation using gemini-3-pro-image-preview with prompts enforcing semantic distillation and flat geometric style, and (2) human verification where three annotators select the best of $k=4$ candidates per batch.

Metric: The signed literal bias is: $$b(t) = S(v_{\text{lit}}, t) - S(v_{\text{id}}, t)$$ and the Semantic Alignment Gap (strength, direction-agnostic) is: $$\Delta(t) = |b(t)|$$ where $S(v, t)$ is instantiated three ways depending on model access: - Discriminative (CLIP-style): cosine similarity in joint embedding space, $S_{\text{disc}}(v,t) = \frac{e_v \cdot e_t}{\|e_v\|\|e_t\|}$ - Open-generative (white-box): Likelihood of Idiomatic Distinction (LID) via next-token logits, $S_{\text{open}}(v,t) = \frac{\exp(\ell_{\text{Yes}})}{\exp(\ell_{\text{Yes}}) + \exp(\ell_{\text{No}})}$ - Proprietary (black-box): self-reported confidence $\gamma \in [0,100]$ normalized to $[0,1]$, validated against behavioral forced-choice frequency ($k=10$ trials; Spearman $\rho \approx 0.79$–$0.82$)

Models are evaluated under two matched conditions per NC: original ADMIRE high-fidelity images (Photo) and DIVA iconographic images (Icon).

Key Contributions¶

DIVA benchmark: 1,000 sense-controlled iconographic images for 100 English noun compounds (literal and idiomatic paired renderings), released openly with metadata and generation protocol.
$\Delta$ metric: Architecture-agnostic Semantic Alignment Gap with three instantiations (embedding geometry, LID logit probability, extrinsic confidence elicitation), enabling within-paradigm comparisons across discriminative and generative VLMs.
Signed bias $b(t)$: Directional companion to $\Delta$ that separates bias magnitude from direction.
Systematic evaluation of 8 VLMs across three architectural families revealing universal Literal Superiority Bias and the consistent gap-reduction effect of visual simplification.

Results¶

Discriminative models (SigLIP 2, EVA-CLIP-18B, MetaCLIP 2): largest gaps, $\Delta \approx 0.245$–$0.262$ (Photo), reduced to $\approx 0.178$–$0.191$ (Icon); all $b > 0$.
Open-generative models (InternVL3-78B, Qwen2.5-VL-32B, LLaVA-OneVision-7B): lower gaps, $\Delta \approx 0.138$–$0.176$ (Photo), reduced to $\approx 0.089$–$0.122$ (Icon).
Proprietary models (GPT-5, Claude 4.5 Sonnet): smallest gaps overall; GPT-5 drops to $\Delta \approx 0.021$ (Icon) from $0.065$ (Photo); Claude 4.5 Sonnet: $0.072 \to 0.028$.
All reductions are statistically significant (paired Wilcoxon signed-rank, $p < 0.001$).
Sense-specified 5-way selection (Acc@1): Iconographic inputs improve accuracy across all families: Discriminative $42.3\% \to 58.7\%$; Open-Generative $61.8\% \to 74.2\%$; Proprietary $78.5\% \to 91.3\%$.
Model scale alone does not resolve literal bias; increased visual fidelity correlates with weaker symbolic alignment.

Limitations¶

Empirical scope is limited to English idiomatic noun compounds; generalizability to other figurative constructions (metaphors, metonymy) or other languages is not established.
Cross-paradigm $\Delta$ comparisons are not directly valid because $S_{\text{disc}}$, $S_{\text{open}}$, and $S_{\text{conf}}$ operate on different scales.
Self-reported confidence from proprietary models is not guaranteed to be calibrated; behavioral validation ($k=10$ trials) partially mitigates this.
Iconographic images were generated by a single image model (gemini-3-pro-image-preview), introducing potential stylistic biases from that model's priors.
The benchmark covers only 100 unique NCs; coverage of the broader idiomatic NC space is limited.

Relevance to Vision-Language Models¶

This paper directly targets a known weakness of VLMs—compositional and figurative understanding—by providing a controlled diagnostic that isolates the role of visual fidelity from semantic reasoning. The finding that hyperrealistic imagery actively suppresses symbolic understanding (even in frontier models like GPT-5 and Claude 4.5 Sonnet) challenges the implicit assumption that photorealistic training data uniformly improves VLM capabilities. The DIVA benchmark and $\Delta$ metric offer researchers a reproducible tool to track compositional alignment separate from perceptual quality, and the iconographic abstraction framework suggests that training or prompting with visually simplified inputs could be a practical intervention for improving figurative language grounding in VLMs.