SigLIP-HD by Fine-to-Coarse Supervision

🕒 Published (v1): 2026-07-10 15:05 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Why this paper was selected

SigLIP-HD fine-to-coarse supervision; ICLR 2026; Hengshuang Zhao group; directly improves MLLM visual encoding

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SigLIP-HD improves visual representation quality at a fixed standard resolution (\(512^2\) px) by training a student encoder to mimic the multi-scale features of a frozen teacher, eliminating the need for expensive high-resolution tiling at inference. Built on SigLIP 2, it consistently outperforms its baseline across 13 MLLM benchmarks with zero added inference cost. The gain is largest on OCR and fine-grained chart tasks.

Problem

MLLMs increasingly rely on high-resolution input to obtain fine-grained visual tokens, but this forces multiple forward passes, image tiling, and post-processing token compression (resamplers, pixel unshuffle), adding significant compute and system complexity. The paper asks whether the full perceptual capacity at a standard medium resolution (\(512^2\)) has been unlocked before scaling resolution further.

Method

Fine-to-coarse supervision. A frozen pre-trained SigLIP 2-So400m/16-512px teacher encodes an image at two scales (\(512^2\) and \(1024^2\)) using non-overlapping sliding windows. The \(1024^2\) feature map \(F^h \in \mathbb{R}^{C \times 2H \times 2W}\) is bilinearly interpolated to base scale and averaged with the base-scale feature \(F^b\) to form a high-quality target \(F^t\). A trainable student (SigLIP-HD), initialized from the same checkpoint and with identical architecture, encodes only the \(512^2\) image to produce \(F^s \in \mathbb{R}^{C \times H \times W}\) and is optimized with L1 loss: \(\mathcal{L} = \|F^s - F^t\|_1\). No projection heads, auxiliary decoders, or human labels are used. Training runs for 90K iterations on 4.5M raw images from Cambrian-1 (34 hours on 32 A100s).

Key Contributions

• Identifies and formally studies the question of whether medium-resolution perception capability has been fully exploited before scaling to native resolution.
• Proposes a label-free, structure-preserving fine-to-coarse supervision that leaves model architecture and I/O unchanged—plug-in checkpoint replacement.
• Ablates multi-scale teacher configurations, inference strategies (sliding window variants, PE interpolation), feature fusion (interpolate+average vs. pixel unshuffle vs. concatenation), and loss functions, justifying each design choice empirically.
• Demonstrates compatibility with both fixed-resolution and AnyRes (native resolution) MLLM pipelines and across multiple LLM backbones (Vicuna-1.5-7B, Llama-3.2-3B, Qwen2.5-7B).

Results

• LLaVA-NeXT / Vicuna-1.5-7B (encoder unfrozen, Table 5): DocVQA 56.0→59.6 (+3.6), ChartQA 61.6→65.2 (+3.6), HRBench 43.5→48.3 (+4.8), TextVQA 65.8→65.7 (flat), avg 62.8→64.4.
• LLaVA-1.5 / Vicuna-1.5-7B (encoder frozen, Table 5): avg 55.8→57.4; DocVQA +2.5, ChartQA +0.9.
• AnyRes / LLaVA-NeXT (Table 6): ChartQA 63.9→67.4 (+3.5), DocVQA 67.6→69.7 (+2.1), avg 64.8→66.3.
• Llama-3.2-3B (Table 7): ChartQA +4.6, DocVQA +2.6, avg 57.2→58.9.
• Qwen2.5-7B (Table 7): DocVQA 62.5→64.2, avg 65.7→66.2.
• Consistent improvement across all 4 SFT/freeze settings tested; L1 loss marginally outperforms cosine similarity and cosine+smooth-L1 alternatives.

Limitations

• Some visual details that are entirely absent in the downsampled \(512^2\) image cannot be recovered regardless of supervision quality; native-resolution pipelines are still needed for those cases.
• The method is validated only on SigLIP 2 and OpenAI-CLIP encoders; generalizability to other encoder families is not systematically established.
• Optimal multi-scale fusion weights and scale counts are found empirically; no theoretical justification is provided, and more sophisticated search (e.g., learned weights across 3–4 scales) is explicitly left out of scope.
• The gain on tasks that do not benefit from fine-grained features (e.g., MMBench, POPE) is marginal or zero, indicating the method is selectively useful.

Relevance to Vision-Language Models

SigLIP-HD directly addresses how to improve the vision encoder component of MLLMs without increasing inference cost, a central design tension as VLM systems scale. The fine-to-coarse distillation paradigm offers a drop-in upgrade path for any MLLM that already uses SigLIP 2, making it immediately actionable for practitioners. It also challenges the implicit assumption in recent VLM work that resolution scaling is the only lever for fine-grained perception, positioning knowledge distillation from multi-scale teachers as a complementary axis. The ablation study on multi-scale feature fusion strategies is useful reference material for anyone designing vision tower components in VLMs.