Glance2Gaze: Efficient Vision-Language Models from Glance Fusion to Gaze Compression

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Glance2Gaze is a cognitively inspired two-stage visual token compression framework for VLMs that mimics human glance-then-gaze eye movement. It combines a Glance Fusion module (text-aware multi-layer ViT feature aggregation) with a Gaze Compression module (query-guided iterative token reduction within the LLM decoder) to achieve superior performance-efficiency trade-offs. On LLaVA-1.5-7B at 11% FLOPs, it retains 95.6% of full-model performance, outperforming all compared baselines.

Problem

Existing visual token compression methods for VLMs treat token pruning either entirely at the encoder stage or within the LLM decoder, but neither paradigm captures the hierarchical, global-to-local attention structure that characterizes human vision. Encoder-stage methods miss instruction-guided compression; LLM-stage methods (FastV, SparseVLM) rely on full attention matrices incompatible with FlashAttention and apply uniform pruning regardless of decoding depth. No prior method jointly addresses global context enrichment and progressive, depth-aware local compression.

Method

Glance Fusion extracts features from S sampled intermediate ViT layers (default: layers {7, 13, 19, 23} of a 24-layer ViT). For each selected layer, a layer-specific projection maps instruction token embeddings into the visual feature space, and a scaled dot-product cross-attention (visual tokens as queries, instruction tokens as keys) produces per-token correlation scores. These scores are softmax-normalized across layers and used as element-wise weights to produce a single instruction-aware visual token sequence VQ of unchanged length—adding <2.72% total FLOPs.

Gaze Compression embeds within the LLM decoder starting at a predefined layer r (shallow layers are left uncompressed to avoid early pruning sensitivity). A single shared learnable query pool Qs ∈ ℝ^(pr×dt) is maintained across all compression layers; at each layer l≥r, the first pl queries (from a monotonically decreasing schedule P = [pr, ..., pR]) attend over visual tokens from the previous layer via cross-attention with 2D RoPE positional encoding, outputting pl compressed tokens. The token count decays to as few as 2 at the final layer. Sharing queries across layers constrains optimization and implicitly encourages pattern consistency.

Key Contributions

• Cognitively motivated two-stage framework unifying global glance (encoder-side feature fusion) and focused gaze (decoder-side progressive compression).
• Glance Fusion: text-conditioned, layer-adaptive aggregation of hierarchical ViT features without increasing visual token count.
• Gaze Compression: shared-query-pool cross-attention compression embedded inside LLM decoder layers, with depth-aware progressive token reduction schedule.
• Compatibility with FlashAttention (no reliance on full attention matrices for pruning decisions).
• Demonstrated generalization to high-resolution (LLaVA-NeXT, up to 2880 tokens) and video (Video-LLaVA, 2048 tokens from 8 frames) settings.

Results

LLaVA-1.5-7B (10 benchmarks): - At 33% FLOPs: 99.9% of full-model average performance; +3.7% over PDrop, +0.8% over VisionZip. - At 22% FLOPs: 98.7% performance; only 1.2% drop vs. full model; SparseVLM drops 2.7%. - At 11% FLOPs: 95.6% performance; +0.7% over VisionZip, +6.6% over SparseVLM.

LLaVA-NeXT-7B (high-resolution, 7 benchmarks): - At 22%/11%/5% FLOPs: 99.2%/97.6%/94.8% performance; +1.2–1.3% over VisionZip at all settings.

Video-LLaVA (4 video QA benchmarks, 6% FLOPs): - 96.2% of full-model performance; +3.0% avg over VisionZip; FastV collapses to 52.1%.

Inference efficiency (LLaVA-NeXT-7B, ~2.67 TFLOPs vs. 53.83): - 6.39× prefill speedup, 3.15× latency reduction, 3.14× throughput improvement over baseline; fastest among FastV, SparseVLM, VisionZip at same FLOPs.

Glance Fusion ablation (LLaVA-1.5-7B): - +1.0 points on TextVQA, +1.6 on VQAv2 vs. Gaze Compression alone; outperforms Dense Connector and MMFuser across 7B and 13B scales.

Limitations

• The fusion and compression heuristics (layer selection, query pool design, compression schedule) lack grounding in explicit cognitive models; the connection to neuroscience remains motivational rather than mechanistic.
• Compression schedule (starting layer r, endpoint pR=2, exponential decay) requires tuning per backbone and FLOPs target; generalization to architectures beyond the LLaVA family is not demonstrated.
• Gaze Compression introduces four linear projection layers per active decoder layer; parameter overhead at very deep LLMs is not analyzed.
• Evaluation is limited to LLaVA-series backbones; performance on more recent, stronger VLMs (e.g., InternVL2, Qwen2-VL) is untested.

Relevance to Vision-Language Models

Glance2Gaze directly targets the central efficiency bottleneck in VLMs: the quadratic cost of processing long visual token sequences in LLM decoders, which is exacerbated by high-resolution and video inputs. It advances the state of token compression by showing that instruction-aware global feature fusion and depth-aware progressive compression are complementary, offering a principled alternative to flat attention-score pruning. The shared query pool mechanism is particularly notable as it achieves FlashAttention compatibility—a practical prerequisite for deployment—unlike prior LLM-stage pruning methods. This work contributes directly to the efficiency-accuracy frontier for open-source VLM deployment and is directly relevant to anyone studying visual token representation, efficient VLM inference, or multimodal attention mechanisms.