Skip to content

Hawaii: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HAWAII is a knowledge distillation framework that compresses multiple specialized visual expert encoders (SAM, ConvNeXt, EVA-02, Pix2Struct) into a single CLIP vision encoder via hierarchical distillation, eliminating multi-encoder inference cost. It introduces teacher-specific LoRA adapters to prevent cross-teacher knowledge conflicts, plus token importance scoring to prioritize informative visual tokens during distillation. The result is a single-encoder VLM that outperforms LLaVA-1.5 and MoVE-KD across standard benchmarks without additional inference overhead.

Problem

Using multiple vision expert encoders at inference time improves VLM visual understanding but incurs proportional compute and memory costs. Existing multi-teacher KD approaches (e.g., MoVE-KD) use a shared LoRA adapter set for all teachers, causing knowledge conflicts from heterogeneous teacher architectures, training data, and objectives, leading to noisy and suboptimal distillation.

Method

HAWAII distills knowledge from N teachers into a single student (CLIP ViT) via two coupled mechanisms:

MoLA (Mixture of LoRA Adapters): Each FFN layer in the student encoder receives (1) Nₜ teacher-specific LoRA adapters (one per teacher) and (2) Nᵍ general-knowledge LoRA adapters. Two sparse routers (2-layer MLPs with top-1 selection) independently gate which teacher-specific and general-knowledge adapter activates per token per layer. The combined output is F(h) + aᵢᵀ(h) + aⱼᴳ(h).

Hierarchical Knowledge Distillation (HKD): - Coarse-grained (CGKD): Teacher visual tokens are unshuffled to match student resolution, channel-concatenated across all teachers, then compressed by a 2-layer MLP to produce a summarized feature Iᶜᵍᵀ. MSE loss aligns the full student output to this consensus. - Fine-grained (FGKD): Each teacher-specific adapter is activated in isolation to produce IᵢS. Token importance scores sᵢ ∈ ℝ¹ˣᵐ are computed via softmax over the similarity matrix between projected teacher tokens and projected text instructions, then used as per-token MSE weights: Lfg = (1/Nₜ) Σ sᵢ · MSE(IᵢS, IˆᵢT).

Total loss: L = Lgen + 0.5(Lfg + Lcg) + 0.05 Lmb (MoE balance loss included).

Base model: Vicuna-1.5-7B LLM + CLIP vision encoder; teachers: ConvNeXt, EVA-02, (+optionally Pix2Struct or SAM).

Key Contributions

  • MoLA module: teacher-specific LoRA adapters (one per teacher) plus general-knowledge LoRA adapters with independent sparse routers, preventing cross-teacher gradient conflicts during distillation
  • Coarse-grained KD: summarizes collective teacher consensus via concat+MLP and aligns student globally via MSE
  • Fine-grained KD: per-teacher distillation with token importance scores derived from teacher visual token–instruction similarity, focusing distillation on task-relevant tokens
  • Single-encoder inference: no multi-encoder overhead at test time

Results

All against LLaVA-1.5 (Vicuna-1.5-7B) as primary baseline; MoVE-KD is the main KD competitor (same teachers as HAWAII base):

  • VizWiz: 50.0 → 53.9 (+3.9 pp vs. LLaVA-1.5); HAWAII outperforms MoVE-KD (52.3)
  • SQA: 66.8 → 70.5 (+3.7 pp); MoVE-KD: 68.7
  • MMBench: 64.3 → 66.9 (+2.6 pp); MoVE-KD: 66.3
  • MMMU: 34.7 → 36.6 (+1.9 pp)
  • MME: 1510.7 → 1540.2 (+29.5)
  • POPE: 85.9 → 87.3 (+1.4 pp)
  • Average across 10 benchmarks: 61.9 → 63.7 (ablation table)
  • 13B variant (HAWAII-13B): SQA 71.6 → 75.0 vs. LLaVA-1.5-13B; outperforms MoVE-KD-13B on SQA (73.2 → 75.0)
  • LLaVA-Next-7B backbone: HAWAII improves over base LLaVA-Next on VQAText (64.9→65.5), GQA (64.2→65.2), SQA (70.1→72.0), POPE (86.5→87.8), MME (1519→1551.3)
  • Ablation confirms all three components (FGKD, token scoring, CGKD) contribute incrementally; 3 general-knowledge adapters optimal (5 causes slight degradation)

Limitations

  • GQA performance slightly decreases when adding token importance scoring, suggesting the scoring may under-weight general knowledge needed for compositional reasoning
  • Adding Pix2Struct as a 4th teacher degrades VQAText, SQA, and SeedBenchI scores relative to the 3-teacher variant, indicating knowledge redundancy is not fully resolved
  • MME degrades when adding SAM as a teacher (HAWAII‡ vs. HAWAII), attributed to MME requiring common-sense reasoning over task-specific visual features
  • DoRA underperforms standard LoRA in this setting, limiting adapter choice
  • POPE accuracy slightly decreases when scaling to 13B, unexplained

Relevance to Vision-Language Models

HAWAII directly addresses a central efficiency bottleneck in VLMs: multi-expert visual encoding provides perceptual diversity but multiplies inference cost, and HAWAII shows this trade-off can be largely eliminated via post-hoc distillation into the existing single encoder. The MoLA design—separating teacher-specific from general adapters with independent routing—is a practical blueprint for conflict-free multi-teacher KD, applicable to any LoRA-amenable vision backbone. Token importance scoring that conditions on instruction text is also notable: it ties visual feature selection to the linguistic context, a pattern consistent with dynamic token pruning work (e.g., FastV) but applied at the distillation stage rather than inference. For VLM researchers, this work establishes that the LLaVA-1.5 architecture can absorb the complementary strengths of SAM, ConvNeXt, EVA, and Pix2Struct into its existing CLIP encoder with modest LoRA parameter overhead.