Skip to content

Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge

🕒 Published (v1): 2026-06-25 20:19 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

Yong Jae Lee, ICML; LLM→vision cross-modal fine-grained knowledge distillation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LaViD (Language-to-Visual Knowledge Distillation) transfers conceptual knowledge from a text-only LLM to a vision-only student model by prompting the LLM to generate multiple-choice questions (MCQs) that probe semantic distinctions between visual classes, then using the LLM's answer logits as distillation targets. Despite the teacher never seeing images, LaViD consistently outperforms recent VLM-based distillation methods across six fine-grained benchmarks.

Problem

Fine-Grained Visual Classification (FGVC) requires distinguishing subtle inter-class differences that purely visual supervision struggles to encode—models overfit to spurious background correlations rather than semantically meaningful traits. Existing cross-modal distillation either requires paired vision-language inputs or a multimodal teacher (e.g., CLIP, LLaVA, InternVL), both of which impose alignment or data constraints.

Method

Stage 1 — LLM logit extraction (done once per dataset): Given class names and dataset metadata, an LLM (GPT-4o for MCQ generation, Qwen2.5-7B for logit extraction) is prompted to produce \(Q\) multiple-choice questions with \(M\) answer options each. Each question contains an <object> placeholder replaced per class. The LLM's pre-softmax logits over answer tokens are extracted and softmax-normalized, forming a per-class target matrix \(T_c \in \mathbb{R}^{Q \times M}\) — a "conceptual signature."

Stage 2 — Student training: A visual backbone with an auxiliary linear head maps the feature vector \(f(x)\) to \(S(x) \in \mathbb{R}^{Q \times M}\). The total loss is: $\(\mathcal{L} = \mathcal{L}_{\text{sup}} + \lambda \mathcal{L}_{\text{LaViD}}, \quad \mathcal{L}_{\text{LaViD}}(x, y) = \frac{1}{QM}\|S(x) - T_y\|_2^2\)$ The distillation targets are class-level constants (fixed after Stage 1), not instance-level predictions, making Stage 1 cost negligible at training time.

Key Contributions

  • Cross-modality knowledge distillation from a language-only LLM to a vision-only student, requiring no paired data, shared embedding space, or multimodal teacher.
  • MCQ-based elicitation of structured, multi-dimensional semantic supervision that encodes inter-class relational geometry beyond label smoothing or embedding similarity.
  • Demonstrated robustness benefit: LaViD substantially improves worst-group accuracy on Waterbirds by anchoring representations to concept-level attributes rather than spurious background correlations.
  • Modular design: LaViD is additive and combines with existing logit-standardization (LS) methods for further gains.

Results

  • vs. MaKD (InternVL2-8B teacher): LaViD (ResNet-18) exceeds MaKD on CUB (70.15 vs. 68.19), Aircraft (83.22 vs. 80.45), Pets (82.29 vs. 82.06), Cars (88.59 vs. 87.96), Flowers (81.34 vs. 80.18).
  • vs. MLKD (visual RN-50 teacher): LaViD surpasses MLKD on CUB (70.15 vs. 68.76), Aircraft (83.22 vs. 85.35 — MLKD wins here), Pets, Caltech across most student architectures.
  • w/ logit standardization (LS+LaViD, RN-18): CUB 72.46, Caltech 84.17, Flowers 85.05, Aircraft 86.21, Pets 85.09, Cars 91.20 — consistently best or second-best across the table.
  • ViT-B/16 student: LaViD improves CLIP-ViT on CUB (84.65→86.69) and Aircraft (81.41→83.42).
  • Waterbirds worst-group accuracy: Consistently higher than independently trained students across all architectures (specific deltas not fully reported in excerpt).
  • ImageNet WordNet subsets (ResNet-18): LaViD improves across all 9 semantic groups (e.g., Bird: 88.65→90.08, Mammal: 78.55→79.15).
  • LLM teacher quality matters: Qwen2.5 > Gemma > LLaMA; larger models generally better; MCQ generator (GPT-4o vs. Gemini 2.5 Pro) has minimal effect.

Limitations

  • Does not scale to large undifferentiated class sets (e.g., full 1000-class ImageNet); requires grouping into semantically coherent subsets for the MCQ approach to be meaningful.
  • MCQ structure assumes class names carry sufficient semantic signal for LLM reasoning — may degrade for classes with opaque or ambiguous names.
  • Auxiliary head adds minor parameter overhead; the distillation loss weight \(\lambda\) requires tuning.
  • Reliance on a capable frontier LLM (GPT-4o) for MCQ generation, though results are robust to substituting Gemini 2.5 Pro.

Relevance to Vision-Language Models

LaViD directly challenges the assumption that cross-modal supervision requires a multimodal encoder: it shows that a text-only LLM's conceptual structure is sufficient to outperform VLM-based teachers (InternVL2-8B, LLaVA-1.5) for fine-grained visual recognition. This has implications for efficient VLM-to-vision distillation pipelines, suggesting that the bottleneck is structured semantic elicitation rather than visual grounding in the teacher. It also connects to the broader hypothesis that LLMs encode modality-agnostic conceptual representations ("platonic representation hypothesis"), providing empirical evidence that these representations transfer usefully across the language-vision modality gap without alignment supervision.