Skip to content

Hierarchical Pre-Training of Vision Encoders with Large Language Models

🕒 Published (v1): 2026-03-31 18:00 UTC · Source: Arxiv · Venue: CVPR · link

Why this paper was selected

CVPR; hierarchical joint pre-training tightly couples vision encoders with LLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HIVE (Hierarchical Pre-Training of Vision Encoders) proposes injecting cross-attention between intermediate layers of a vision encoder and an LLM during pre-training, so the encoder is explicitly supervised to preserve multi-scale visual features. At inference, the enriched encoder reverts to standard token-concatenation (LLaVA-style), avoiding deployment overhead. The method outperforms self-attention baselines on both image classification and VLM benchmarks.

Problem

Existing VLMs feed flattened final-layer image embeddings into the LLM's input, discarding the hierarchical structure of intermediate encoder features. This shallow integration limits gradient flow back into the vision encoder, prevents the encoder from learning to preserve low-level spatial detail alongside high-level semantics, and leaves multi-scale representations underutilized during pre-training.

Method

HIVE adds hierarchical cross-attention between the vision encoder and LLM during pre-training only. Given image \(I \in \mathbb{R}^{H \times W \times C}\), the encoder produces layer-wise features \(F_l = f_l(F_{l-1})\) for \(l=1,\ldots,L\). A 25% uniform subset \(S \subset \{1,\ldots,L\}\) of encoder depths is selected. A per-layer 2-layer MLP projector \(g_l\) maps each \(F_l\) to the LLM's token dimension: \(T_{\text{LLM},l} = g_l(F_l)\). Cross-attention layers are injected into the LLM at the corresponding depths in \(S\):

\[H_l = \text{CrossAttention}(Q_l, K_l, V_l), \quad A_l = \text{Softmax}\!\left(\frac{Q_l K_l^\top}{\sqrt{d}}\right)\]

where \(Q_l\) comes from the LLM's hidden state at layer \(l\) and \(K_l, V_l\) are projected from \(T_{\text{LLM},l}\). Training uses next-token prediction loss (\(\mathcal{L} = -\sum_t p_t \log \hat{p}_t\)) via a three-stage curriculum: (1) projector-only training with frozen encoder and LLM; (2) joint projector + LLM training with frozen encoder; (3) full end-to-end training. After pre-training, the cross-attention layers are discarded and downstream fine-tuning uses the standard LLaVA architecture, so no overhead is incurred at deployment. Complexity during pre-training is \(O(L_l L_s d^2 + L_l N_t d^2)\) with \(L_s \ll N_v\), vs. \(O(L_l N^2 d/2 + L_l N d^2)\) for full self-attention over joint vision+text tokens.

Key Contributions

  • Hierarchical cross-attention mechanism that routes intermediate vision encoder features to aligned LLM layers, forcing the encoder to maintain multi-scale representations under language modeling supervision.
  • Three-stage progressive training strategy adapted for hierarchical cross-attention to avoid catastrophic forgetting and gradient explosion.
  • Deliberate architecture shift at inference (cross-attention during pre-training only; standard token concatenation downstream), transferring representational benefit without deployment cost.
  • Demonstrated gains on both classification (CIFAR-10/100, ImageNet-1K, Tiny-ImageNet, Food-101, Stanford Cars, Oxford Pets, Caltech-256) and VLM tasks (MME, GQA, OK-VQA, ScienceQA).

Results

  • Classification: HIVE consistently outperforms the self-attention baseline (SA) and base CLIP/SigLIP across all eight classification benchmarks; largest gains on fine-grained datasets (Food-101, Caltech-256, Oxford Pets with SigLIP backbone). Results reported as mean ± SD over 3 runs (exact numbers in Table 1, truncated from provided text).
  • VLM benchmarks (SigLIP-based models, LLaVA fine-tuning with Llama-3.2-1B-Instruct): HIVE outperforms SA on MME, GQA, OK-VQA, and ScienceQA.
  • All experiments run on a single RTX 3090 (24 GB); pre-training LLM backbone is MobileLLM-350M.
  • Specific numeric results were in Table 1 and downstream VLM tables, which were truncated in the provided text.

Limitations

  • Experiments conducted on a single RTX 3090 with MobileLLM-350M as the pre-training LLM backbone; scalability to larger LLMs and multi-GPU settings is not demonstrated.
  • Only SigLIP-based models are evaluated on VLM benchmarks (CLIP-based VLM results omitted to reduce compute), limiting breadth of comparison.
  • The three-stage warm-up is not ablated due to hardware constraints; the authors acknowledge omitting exhaustive grid searches.
  • Exact numeric results for the classification table were not fully reported in the provided text excerpt.
  • Cross-attention overhead during pre-training adds cost not quantified in wall-clock terms.

Relevance to Vision-Language Models

HIVE directly addresses a core architectural bottleneck in VLMs: the under-utilization of intermediate encoder representations when integrating vision into LLMs. By using hierarchical cross-attention as a pre-training regularizer rather than a deployment mechanism, it sidesteps the inference-cost argument typically leveled against cross-attention architectures (cf. Flamingo, BLIP-2). This is relevant to researchers tracking VLMs because it offers a principled route to richer vision encoders that remain compatible with efficient LLaVA-style deployment pipelines. The work also complements ongoing efforts (AIMv2, SigLIP) to improve vision encoder pre-training objectives by showing that LLM feedback through intermediate cross-attention is a viable and compute-tractable signal.