ENTRAP-VL: A Taxonomic Probe for Dual Contextual Entrainment in Vision-Language Models

🕒 Published (v1): 2026-07-22 12:47 UTC · Source: Arxiv · link

Why this paper was selected

Taxonomic probe showing VLMs reliably succumb to irrelevant contextual entrainment

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ENTRAP-VL is a manually curated benchmark of 1,500 items designed to probe contextual entrainment — the tendency of models to be pulled toward auxiliary context regardless of its relevance or truth — in vision-language models (VLMs). The paper argues that this phenomenon is inherently dual in VLMs (separately drivable by textual and visual context), introduces a two-axis taxonomy capturing this duality, and provides the instrument and protocols without reporting model results.

Problem

Contextual entrainment — the systematic elevation of context-present tokens' probability independent of relevance, as identified by Niu et al. in unimodal LMs — has not been rigorously studied in VLMs. Existing distraction benchmarks and interpretability work measure adjacent phenomena, and templatic unimodal benchmarks cannot be ported to the multimodal setting because: (1) VLMs have two independent channels (textual and visual context) that can pull outputs in different directions; and (2) perceptual grounding introduces a contradictory veracity level (false of the depicted scene yet possible in the world) with no unimodal analog.

Method

A two-axis taxonomy structures all context conditions: - Association axis: relatable (entity/property present in or associated with the depicted scene) vs. random (no semantic association). Relatability is defined by the perceptual scene, not by nominal category membership — these can come apart. - Veracity axis: true (holds of the scene), contradictory (false of this scene but world-plausible), counterfactual (false in the world).

The dataset is split into two streams: - Textual-entrainment stream (800 items): the query concerns a fixed image; the context is textual. Supports all eight conditions: relatable_true, relatable_contradictory, relatable_distractor, random_distractor, relatable_distractor_short, random_distractor_short, relatable_counterfactual, random_counterfactual. Each item carries three statements per condition. - Visual-entrainment stream (700 items): the query is textual; the context is visual. The veracity axis does not apply to photographs (an image is a perceptual referent, not a proposition), so only the association axis instantiates — yielding three conditions.

Every short-form trigger (bare entity names) is constructed to be a wrong answer, so any pull toward it is cleanly attributable to entrainment rather than coincidental correctness. Curation is manual and item-aware; templatic generation is explicitly rejected because conditions such as "a statement that contradicts this image" require per-item human judgment.

Key Contributions

• Position argument that VLM contextual entrainment is a distinct, dual phenomenon requiring a purpose-built instrument, distinct from distraction benchmarks and unimodal entrainment probes.
• Two-axis taxonomy (association × veracity) formalizing eight context conditions, including the relatable_contradictory condition that is inexpressible in the unimodal setting.
• ENTRAP-VL: 1,500 manually curated items across eight image categories, split into textual-entrainment (800 items) and visual-entrainment (700 items) streams; to be released publicly on HuggingFace.
• Evaluation protocols describing how the dataset supports controlled measurement, including with/without-context contrasts adapted from unimodal context-aware decoding.

Results

No model evaluation results are reported. The paper is explicitly a position and resource contribution; it provides the instrument for the community to conduct the empirical investigation, not empirical findings itself.

Limitations

• No empirical model results: the paper makes no claim about whether any VLM does or does not exhibit dual contextual entrainment.
• The visual stream cannot instantiate the veracity axis; images are world-knowledge-consistent by construction, so contradictory and counterfactual conditions have no visual analog.
• The random_contradictory cell is structurally absent (a contradiction of something not in the scene is incoherent), leaving the taxonomy non-factorial by design.
• Dataset is restricted to behavioral evaluation; authors explicitly warn against using it for fine-tuning, which would contaminate the probe.
• The four-axis contrast between relatable_distractor_short and random_distractor_short isolates relatability in bare-trigger form but does not cover all possible form/content combinations.

Relevance to Vision-Language Models

ENTRAP-VL directly targets a subtle failure mode in VLMs deployed in RAG-augmented VQA and similar pipelines, where auxiliary context (retrieved passages, captions, or images) can silently bias outputs irrespective of relevance. The benchmark formalizes a veracity distinction — contradictory (scene-false but world-plausible) vs. counterfactual (world-false) — that is unique to perceptually grounded models and unavailable to text-only benchmarks, providing a sharper lens than prior visual-distraction work. By separating textual and visual entrainment streams through design rather than post-hoc analysis, it enables controlled cross-channel comparisons unavailable in existing VLM robustness benchmarks. This fills a clear gap between mechanistic interpretability work on attention heads and behavioral robustness evaluation, offering the community a structured stimulus set for probing how multimodal context shapes VLM outputs.