Skip to content

Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter It

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VL training consistently improves LM performance on a text-only taxonomic QA benchmark (TaxonomiGQA), but not because it restructures underlying taxonomic knowledge. Instead, VL training improves how models deploy existing taxonomic knowledge in task contexts, as evidenced by representational analyses showing better linear separability of hypernym vs. non-hypernym question representations in VLMs.

Problem

Prior work on how VL training affects LMs' linguistic representations has yielded mixed or marginal results. No prior work had specifically compared VLM–LM minimal pairs on taxonomic knowledge—a domain where perceptual grounding plausibly has a direct effect on lexical-conceptual organization.

Method

The authors construct TaxonomiGQA, a text-only QA dataset derived from GQA: scene graphs are converted to text descriptions, object mentions in questions are substituted with WordNet hypernyms (e.g., "bear" → "mammal" → "vertebrate" → "animal"), and four negative-sample questions per instance replace the target with non-hypernyms. The final dataset has 148,020 instances over 276 hyponym-hypernym pairs and 126 chains. Seven VLM–LM minimal pairs (e.g., Qwen2.5-7B-Instruct vs. Qwen2.5-VL-Instruct, Llama-3.1 vs. MLlama-3.2) are evaluated with three metrics: Overall Accuracy, Conditional Accuracy (conditioned on answering the unsubstituted question correctly), and Hierarchical Consistency (HC, requiring correctness on the full chain).

To discriminate between H1 (VL training alters taxonomic knowledge) and H2 (VL training improves deployment), the authors run: - TAXOMPS: direct "Is C1 a C2?" elicitation of taxonomic membership judgments - RSA on Park et al.'s [52] hierarchically-sensitive unembedding space: compares VLM, LM, and WordNet pairwise structure via Spearman correlation - Static embedding cosine similarity: tests whether hypernym-hyponym pairs are more similar in VLM embeddings than in LM embeddings - Layerwise logistic regression: uses contextualized similarity between hyponym (in scene description) and hypernym (in question) to predict correctness; compares odds ratios across layers - PCA + SVM: projects last-layer hidden states of questions with hypernym vs. negative substitutions into 2D PCA space; fits soft-margin SVM to measure linear separability

Visual similarity hypothesis is tested via cosine similarity of vision-encoder embeddings (from THINGS dataset) as a predictor of VLM conditional accuracy in a linear mixed-effects model.

Key Contributions

  • TaxonomiGQA: a text-only taxonomic QA benchmark (148,020 instances) with hierarchically structured metrics anchored on model success at prerequisite tasks
  • TAXOMPS: a direct taxonomic membership elicitation dataset (1,380 questions) enabling clean comparison of VLM–LM pairs on taxonomic judgment per se
  • Empirical finding across 7 VLM–LM pairs that VLMs consistently outperform LMs on text-only taxonomic QA despite no fundamental change to underlying taxonomic representations
  • Representational evidence (RSA, PCA/SVM) localizing the VLM advantage to deployment: improved linear separability of question representations encoding taxonomic vs. non-taxonomic relations
  • Preliminary causal account linking VLM gains to visual cohesion of hypernym categories (β = 0.52, SE = 0.19, p < .01; effect absent for LMs)

Results

  • On TaxonomiGQA (all three metrics), VLMs outperform their LM counterparts in 6 of 7 pairs; most salient gap in Qwen2.5-7B-Instruct vs. Qwen2.5-VL-Instruct
  • On TAXOMPS (direct taxonomic elicitation), VLM–LM pairs perform nearly identically and well, except Llama-3.1 vs. MLlama-3.2
  • RSA of Park et al.'s hierarchically-sensitive unembedding space: VLM–LM similarity ≥ 0.95 for all pairs; VLM–WordNet and LM–WordNet RSA scores are essentially equal (e.g., 0.38–0.43 for both within each pair)
  • Static embedding similarity: no VLM–LM pair shows VLMs encoding hypernym-hyponym pairs significantly more similarly (the one significant effect for Qwen2.5 is in the opposite direction)
  • PCA/SVM: SVM classification error on VLM question representations = 0.36 vs. LM = 0.52 (Qwen2.5 pair), indicating substantially better linear separability of taxonomic distinctions in VLMs
  • Visual similarity mixed-effects model: β = 0.52 (p < .01) for VLM conditional accuracy; β = 0.23 (p = 0.18) for LM, suggesting visual cohesion specifically drives VLM gains

Limitations

  • No causal evidence linking representations to behavior; causal analysis would require access to training data and objectives, which is infeasible at scale for open models
  • Confound: VL-trained models see additional text during VL training; the text-only portion of VL training data could independently account for improvements
  • SVM separability analysis captures only linear encoding of taxonomic distinctions; non-linear separability is not tested
  • Minimal pair selection is constrained by publicly reported base LM provenance; some pairs (e.g., MLlama-3.2 vs. Llama-3.1) differ substantially in parameter count
  • Visual similarity hypothesis is preliminary and tested only on one VLM pair (Qwen2.5-VL-I)

Relevance to Vision-Language Models

This paper directly probes what VL training actually changes in the language backbone of VLMs—a question with practical consequences for model design (freeze vs. update the LM during VL training) and training data selection. The finding that VL training improves knowledge deployment rather than knowledge content challenges naive interpretations of VLM superiority on linguistic tasks and complements the Platonic Representation Hypothesis by showing that cross-modal training can induce non-trivial functional changes even when representational geometry is largely preserved. For researchers tracking VLMs, it establishes TaxonomiGQA as a sensitive diagnostic for deployment-level capabilities and motivates investigation into how visual grounding restructures contextual inference pathways rather than static lexical representations.