Cross-modal Associations in Vision and Language Models: Revisiting the Bouba-Kiki Effect¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper re-evaluates whether CLIP (ResNet-50 and ViT variants) exhibits the bouba-kiki effect — the human tendency to associate pseudowords like "bouba" with round shapes and "kiki" with jagged ones. Using probability-based probing and a novel Grad-CAM interpretability approach closely mirroring human experimental paradigms, neither CLIP variant shows consistent, human-like cross-modal shape-word associations. The findings contradict prior claims and suggest CLIP's contrastive training objective does not instill grounded sensory form-meaning mappings.

Problem¶

Prior work on the bouba-kiki effect in VLMs produced conflicting results: Alper & Averbuch-Elor (2023) claimed strong evidence for the effect in CLIP/Stable Diffusion using embedding similarity, while subsequent studies (Verhoef et al. 2024; Iida & Funakura 2024; Loakman et al. 2024) found weak or absent effects. The embedding similarity method may detect language-specific lexical regularities rather than true cross-modal sensory mappings. No prior work applied interpretability tools to probe why models succeed or fail, and direct comparison with human psycholinguistic data on identical stimuli was lacking.

Method¶

Two CLIP variants (ResNet-50 and ViT-B) are probed on controlled jagged/round image pairs drawn from prior human studies and newly generated stimuli (random points connected by curved vs. straight segments).

Experiment 1 — Probability probing: For each image, the model assigns probabilities to all candidate labels (pseudowords embedded in 10 prompt templates); the highest-probability label is taken as the model's preference. Four pseudoword label sets are used: classic bouba/kiki and maluma/takete pairs; English adjectives (sharp/curved synonyms, as a sanity baseline); phonologically constructed two-syllable pseudowords following Nielsen & Rendall (2013) sonorant/plosive × rounded/non-rounded vowel logic (162 "pure" labels); and pseudowords generated per the Alper & Averbuch-Elor (2023) three-syllable method. A prediction is scored congruent only when both shapes in a pair are matched correctly (following Ćwiek et al. 2022 criterion).

Experiment 2 — Grad-CAM visual attention: Concatenated image pairs (curved + jagged side-by-side, n=17 pairs) are used as input; Grad-CAM gradients from the final convolutional layer (ResNet) or [CLS]-to-patch attention (ViT) identify which region receives the most attention for a given label. The proportion of trials where the model attends more to the expected region is measured across prompts and label types. Bayesian regression models (brms) are used throughout for inference.

Key Contributions¶

Comprehensive re-evaluation of the bouba-kiki effect in two CLIP architectures using paradigms directly matched to human psycholinguistics, enabling quantitative human–model comparison.
Novel application of Grad-CAM to cross-modal association testing, revealing where models attend during shape-word matching — not just what they predict.
Demonstrates that prior positive results (embedding similarity method) likely capture language-specific lexical regularities rather than genuine perceptual cross-modal mappings.
Shows individual modalities are separable by CLIP (English adjectives correctly matched), isolating the failure to the cross-modal integration step rather than unimodal encoding.

Results¶

English adjective baseline: Both models match adjectives (e.g., "sharp", "curved") to images at ~80% — above chance, confirming basic shape discrimination works.
Pseudoword probability probing: Neither ResNet nor ViT exceeds chance consistently for any pseudoword set. Descriptive means are at chance level across prompts and word types; only isolated prompt–model–word-type combinations occasionally cross chance.
Grad-CAM, bouba/kiki + maluma/takete pairs: Neither model reliably exceeds chance (25%, as both labels must be matched correctly). Human performance (Ćwiek et al. 2022) is substantially above model performance.
Grad-CAM, all pseudowords: ResNet shows a global bias toward attending to round shapes regardless of label category, precluding a bouba-kiki effect. ViT matches English adjectives reliably but collapses to chance for pseudowords.
Attention patterns: Grad-CAM heatmaps predominantly focus on shape centres or background regions, not on the edge/curvature features that would drive a bouba-kiki response; ViT background artifacts replicate Darcet et al. (2024).
Model consistency: ResNet is consistent in its predictions across target locations for 77.0% of trials; ViT for 73.5% — consistently wrong rather than randomly wrong.

Limitations¶

Only CLIP (ResNet-50 and ViT) is tested; proprietary models (GPT-4o, Gemini) and larger multimodal architectures with alignment layers are excluded, leaving open whether those layers could recover cross-modal associations.
Study uses text-based pseudowords; the original bouba-kiki effect is auditory. Though prior work suggests tight orthographic–phonological coupling makes this unlikely to matter substantially, audio-visual models are not tested.
Sentence-level contrastive loss in CLIP conflates the cross-modal test with the model's natural objective; sub-word tokenization may occasionally fragment phonologically meaningful pseudoword structure (though authors argue this cannot explain the overall null result).
Image stimuli cover a limited range of shape variation; generalization to other shape properties is untested.

Relevance to Vision-Language Models¶

This paper is directly relevant to understanding the limits of VLM cross-modal grounding: it shows that CLIP — the frozen ViT backbone underlying BLIP2, LLaVA, InternVL, Molmo, and others — does not internalize human-like perceptual form-meaning associations, suggesting that alignment layers in downstream architectures inherit this representational gap. The negative result challenges the assumption that statistical co-occurrence in large-scale image-text data is sufficient to produce cognitively grounded multimodal representations, pointing instead to the absence of embodied, phonosemantic learning signals. Methodologically, the Grad-CAM probing approach offers a reusable interpretability-based diagnostic for testing how VLMs attend during multimodal association tasks, complementing behavioral benchmarks. For researchers tracking VLMs, this situates bouba-kiki as a useful minimal probe of cross-modal grounding quality — one that current architectures consistently fail at — with implications for spatial reasoning, compositional understanding, and human-AI communication.