Skip to content

Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialogue

🕒 Published (v1): 2026-06-30 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLMs evaluated as overhearers of the HCRC MapTask corpus systematically conflate potential common ground (co-presence of landmarks on maps) with established common ground (interpretations confirmed through dialogue interaction). Providing map images or textual map descriptions both trigger the same over-alignment bias, while content-free visual controls (blank/shuffled maps) suppress alignment predictions entirely, confirming the bias is content-driven, not modality-driven.

Problem

Grounding in dialogue is incremental: mutual understanding must be negotiated, not simply inferred from shared perceptual access. Existing VLM evaluations do not disentangle what could be shared (co-present referents) from what has been shared (interpretations established through dialogue acts). No prior work tested VLMs as third-party overhearers of information-asymmetric dialogue with perspectivist ground truth distinguishing each participant's landmark interpretation.

Method

The authors formulate interpretation matching as a binary classification task over 13,077 reference expressions (REs) drawn from perspectivist annotations of the HCRC MapTask corpus (72.1% aligned, 27.9% not aligned). Each instance presents a VLM with a dialogue excerpt (four context-window sizes: curL, curT, startL, startT) and map information under seven modality conditions (text-only; authentic giver/follower/both map images; textual landmark-name lists; textual discrepancy descriptions; blank/shuffled maps as non-informative visual controls). The model produces a constrained YES/NO token via vLLM logit masking (temperature 0). Primary model is Qwen3-VL-8B-Instruct; four additional models (Qwen3-VL-2B/4B, Gemma3-4B/12B) are evaluated on a baseline grid. Analysis includes Expected Calibration Error (ECE) split by gold label, per-grounding-status accuracy (aligned / pending / misunderstood), and reference-chain tracking by chain length.

Key Contributions

  • An evaluation methodology for probing VLMs' ability to distinguish potential from established common ground in information-asymmetric situated dialogue.
  • Empirical evidence that the over-alignment bias is content-driven: textual map descriptions reproduce it, non-informative images suppress it.
  • Calibration and reference-chain analyses revealing that models are confidently wrong on gold-NO instances under map conditions (\(\text{ECE}_{\text{no}}\) rises from .235 to .524 follower-only vs. .235 text-only), pointing to static cue reliance rather than grounding-process tracking.
  • Status-level decomposition showing the trade-off: map access boosts aligned accuracy (+23–29 pp) while sharply degrading pending accuracy (−17–27 pp).

Results

  • Best overall F1 (Qwen3-VL-8B, startT, both maps): \(F1_{\text{macro}} = .671\) vs. \(.591\) text-only.
  • Over-alignment under map access: yes-rate shifts from .515 (text-only) to .727 (both maps), with \(\text{recall}_{\text{pos}}\) rising .590→.822 and \(\text{recall}_{\text{neg}}\) dropping .677→.518.
  • Single-map amplifies bias further: giver-only yes-rate .791, \(\text{recall}_{\text{neg}}\) .436; follower-only yes-rate .794, \(\text{recall}_{\text{neg}}\) .408.
  • Textual descriptions reproduce bias: landmark-name list (\(F1 = .636\), yes-rate .675); discrepancy detail (\(F1 = .668\), yes-rate .716) — close to real maps (.727).
  • Non-informative controls avoid bias: blank maps yes-rate .184, shuffled maps .193 — well below text-only baseline (.515).
  • Calibration: map conditions achieve low \(\text{ECE}_{\text{yes}}\) (.057–.094) but high \(\text{ECE}_{\text{no}}\) (.403–.524) versus near-symmetric ECE under text-only (.263/.235).
  • Status-level: pending accuracy drops .691→.419–.523 under all map conditions; misunderstood accuracy drops to .255 (follower-only).
  • Cross-model: Qwen3-VL-8B shows the bias clearly; Qwen3-VL-4B shows the opposite (maps make it more conservative); Gemma3-12B is extremely conservative throughout (yes-rate ≤.231).

Limitations

  • Evaluation is restricted to 2B–12B parameter instruction-tuned open-source models; larger or proprietary models are not tested.
  • The corpus is English-only and domain-specific (route-following navigation), limiting generalisability to other collaborative or situated dialogue genres.
  • The VLM role is strictly that of an overhearer (no interactive grounding); results may not transfer to models operating as dialogue participants with repair capabilities.
  • The class imbalance (72.1% aligned) is controlled for via macro-F1 and per-class recall but may still interact with model-specific calibration defaults.
  • Cross-model generalisation is assessed only on a baseline condition grid, not the full modality grid.

Relevance to Vision-Language Models

This paper directly probes a failure mode in VLMs' pragmatic reasoning: the inability to distinguish perceptual co-presence from communicatively established reference, a distinction that requires tracking dialogue dynamics rather than static scene content. It operationalises the overhearer illusion as a measurable VLM bias, providing a reproducible benchmark (MapTask perspectivist annotations) and a modality-ablation design that can transfer to other grounded-dialogue settings. For researchers working on VLM evaluation or multimodal grounding, the key methodological finding — that textual and visual map representations trigger identical biases — constrains claims about multimodal reasoning being qualitatively distinct from text-only reasoning. It also surfaces calibration asymmetry as a diagnostic tool: confidently wrong predictions on non-aligned cases identify where static referential cues override dialogue-level evidence.