Skip to content

ToVE: Efficient Vision-Language Learning via Knowledge Transfer from Vision Experts

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💎 Ask ChatGPTâœĶ Ask Claude

TL;DR

ToVE is a framework for data-efficient vision-language learning that routes knowledge from a hub of pre-trained vision experts (depth, normals, edges, DINO, EVA) into CLIP vision tokens via a token-aware gating network and residual addition. It achieves competitive performance using ~3M image-text pairs — two orders of magnitude fewer than comparable VLMs. A "knowledge merging" step subsequently distills all expert knowledge back into a single CLIP encoder, eliminating expert overhead at inference.

Problem

Standard VLMs require billions of image-text pairs to develop fine-grained visual perception (spatial reasoning, object discrimination, hallucination resistance). CLIP-based efficient VLMs like Prismer still exhibit weak spatial reasoning and hallucination on small datasets (e.g., Prismer-Z drops −7.9% VSR when data scales from 12.7M → 3M pairs). The gap: no mechanism to inject rich, domain-specific visual priors without large-scale pre-training.

Method

Framework: Frozen CLIP ViT encodes vision tokens. A hub of K heterogeneous pre-trained experts (3 low-level: depth, surface normal, edge; 2 embedding: DINO, EVA) process the same image in parallel.

Token-aware gating: A lightweight gating network G takes each vision token t_i as input and outputs a softmax-normalized weight vector over K experts. All K experts contribute (soft ensemble, not top-1 MoE), with additive Gaussian noise for exploration.

Residual knowledge transfer: Expert tokens are projected to a common embedding space via expert-specific MLP first layers + shared second layer, then fused into each vision token as: tĖƒ_i = t_i + Îŧ × MLP(t_exp_i) The scalar Îŧ (kept small, e.g., 0.1) ensures CLIP's generalizability is preserved. A cosine-scheduled load-balancing loss Laux prevents gating collapse during early training.

Expert detachment: Experts with low average ensemble weights can be removed post-training (gating scores set to −∞) without retraining, providing a compute-accuracy trade-off knob.

Knowledge merging (ToVE-Lite): After transfer training, the CLIP encoder is fine-tuned to minimize L2 distance between its raw tokens and the knowledge-transferred tokens, jointly with language modeling loss. Inference then requires only the merged CLIP — no expert models needed.

Key Contributions

  • Token-aware gating network that assigns per-token soft ensemble weights over heterogeneous vision experts
  • Residual knowledge transfer strategy (Îŧ-scaled addition) that integrates expert knowledge without degrading CLIP generalizability
  • Pluggable expert detachment: low-contribution experts can be removed at inference without retraining
  • Knowledge merging via joint LM + L2-norm distillation that collapses all expert knowledge into a single CLIP encoder

Results

  • Zero-shot NoCaps captioning (3M pairs): ToVE CIDEr 110.2 / SPICE 14.9 vs. PrismerBASE (12.7M pairs) CIDEr 87.5 — +22.7 CIDEr with 4× less data; ToVE-Lite (no expert inference) achieves CIDEr 104.1
  • Fine-tuned NoCaps: ToVE CIDEr 113.1 vs. BLIP (14M) 102.4, GIT (10M) 109.1
  • Fine-tuned COCO captioning: ToVE BLEU-4 40.3 / SPICE 24.5, competitive with BLIP (129M pairs)
  • VQAv2 (test-dev/test-std): ToVE 75.4/75.8 vs. MAMO (4M) 76.1/76.2 — comparable
  • VSR (spatial reasoning, 3M): ToVE 67.7% vs. Prismer-Z (3M) 55.3% (+12.4%) and vs. Prismer-Z (12.7M) 63.2%
  • POPE-A (hallucination, 3M): ToVE 82.5 vs. Prismer-Z (3M) 80.5 (+2.0%)
  • ToVE-Lite retains 95.6% of full ToVE zero-shot CIDEr after knowledge merging

Limitations

  • Expert hub is fixed at five pre-selected models (depth, normal, edge, DINO, EVA); no principled strategy for selecting or scaling the expert set to new domains
  • Low-level experts use randomly initialized CNN encoders for patch embedding, which may underutilize their full representational capacity
  • Model collapse observed when all experts are removed during pluggable-expert ablation, indicating the adapted language model input space diverges from bare CLIP tokens — limits portability
  • Evaluated only on base-size architectures; scaling behavior to larger CLIP/LM backbones is not studied
  • Knowledge merging still requires an additional fine-tuning stage with expert inference active

Relevance to Vision-Language Models

ToVE directly addresses a core bottleneck in VLM development: the reliance on massive image-text corpora to build visual perception capabilities. By treating the CLIP visual backbone as augmentable via external domain-specific experts and then collapsing that augmentation into the encoder itself, it offers a data-efficient path to stronger spatial reasoning and reduced hallucination — two well-documented weaknesses of CLIP-based VLMs. The token-level routing mechanism is a principled alternative to auxiliary vision token approaches (e.g., Prismer's Resampler), making the knowledge transfer explicit and interpretable. For researchers tracking VLMs, this connects to threads on MoE-in-vision, knowledge distillation, and modular multimodal architectures.