Evaluation of Multilingual Ability to Use Spatial Deictic Expressions in Vision-Language Models

🕒 Published (v1): 2026-07-08 10:34 UTC · Source: Arxiv · Venue: ACL · link

Why this paper was selected

ACL; multilingual spatial deictic eval; exposes systematic VLM spatial reasoning failure mode

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces the first multilingual benchmark for evaluating whether VLMs can use spatial deictic expressions (demonstratives like "this"/"that") in a human-like manner across Japanese, Korean, English, and Chinese. Experiments on four open VLMs reveal that all tested models fail to shift demonstrative choice with object distance the way humans do, particularly under-using distal forms in three-demonstrative languages. The benchmark is grounded in the psycholinguistic "memory game" paradigm from Coventry et al. (2023).

Problem

Prior VLM spatial-reasoning benchmarks are almost exclusively English-only and do not evaluate spatial deixis—context-dependent expressions (demonstratives) whose correct use requires jointly grounding visual distance cues with language-specific distance semantics. Cross-linguistic variation in demonstrative systems (2-way in English/Chinese vs. 3-way proximal/medial/distal in Japanese/Korean, with differing medial semantics) and inherent ambiguity in distance categorization make this a distinctive, unstudied challenge.

Method

The authors adapt the psycholinguistic "memory game" (Gudde et al., 2018; Coventry et al., 2023) into a VQA benchmark. Using Blender, they render 60 synthetic first-person images of a disk with a colored shape on a long desk at three distances (0.25 m, 1.50 m, 2.75 m), corresponding to proximal/medial/distal regions from Coventry et al. Models are prompted in each of four languages to produce exactly "[Demonstrative] [Color] [Shape]". Demonstrative probabilities are computed from output logits via softmax over the target demonstrative tokens. Fidelity to human usage is measured as the Jensen-Shannon distance \(\text{JSD}(P \| Q)\) between the model's distribution and the human distribution reported in Coventry et al. (2023), averaged across distances. Models evaluated: Gemma 3 4B, Gemma 3 12B, Qwen3-VL 8B, Qwen3-VL 32B.

Key Contributions

• First benchmark specifically targeting VLM demonstrative/spatial-deixis use, grounded in a validated psycholinguistic paradigm.
• Multilingual coverage (Japanese, Korean, English, Chinese) exposing cross-linguistic variation in demonstrative systems.
• Quantitative comparison of VLM demonstrative distributions against human data via Jensen-Shannon distance.
• Empirical finding that VLMs uniformly fail to shift demonstrative choice as a function of object distance, especially suppressing distal forms in three-demonstrative languages.
• Correlation analysis (\(r = -0.40\)) between object recognition accuracy and distributional fidelity to human demonstrative usage.

Results

• All four models fail to reproduce the human distance-dependent demonstrative shift; distal demonstratives in Japanese and Korean remain below 5% probability for most models regardless of distance.
• Gemma 3 4B performs worse than a uniform distribution baseline in all languages (average JS distance: Japanese 0.608, Korean 0.871, English 0.504).
• Qwen3-VL 32B is closest to humans in Japanese (avg JS = 0.506) and Chinese (avg JS = 0.373), still worse than or comparable to uniform in Korean (avg JS = 0.745).
• Qwen3-VL 8B achieves the highest recognition accuracy (50/60 images in English), while Gemma 3 4B is lowest (12/60 in Japanese and Chinese).
• English JS distances are consistently lower than Korean across all models; Korean is the hardest language for all tested models.
• Modest negative correlation (\(r = -0.40\)) between image recognition accuracy and JS distance from humans across 15 experimental settings.

Limitations

• Only 60 images total (5 shapes × 4 colors × 3 distances); limited scene diversity.
• Only four open models evaluated; closed models (e.g., GPT-series) excluded due to inability to obtain raw logits.
• Constrained output format (fixed three-word template) may not reflect naturalistic demonstrative use.
• Prompts machine-translated via Gemini with manual correction only for missing medial demonstratives; translation quality not systematically validated.
• Human comparison data sourced from a single prior study (Coventry et al., 2023) rather than newly collected human baselines.
• Paper text is truncated before the Limitations section is completed.

Relevance to Vision-Language Models

This work directly probes a blind spot in VLM spatial reasoning: the grounding of context-dependent, language-specific linguistic forms (demonstratives) in visual distance information. It extends spatial-reasoning evaluation beyond English and beyond object-relation tasks (left/right/front/behind) to the deictic layer of language, exposing that current VLMs lack distance-sensitive reference frame internalization. The multilingual design is particularly relevant to the broader VLM community because it reveals that high English performance does not transfer to typologically similar tasks in lower-resource languages like Korean. The benchmark methodology—synthetic Blender scenes paired with logit-level distributional analysis against psycholinguistic human data—offers a replicable template for future multimodal, multilingual pragmatics evaluation.