VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions

🕒 Published (v1): 2026-03-24 17:58 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; sparse dynamic VL interactions for LVLM efficiency without information bottleneck

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VISOR improves inference efficiency of Large Vision-Language Models (LVLMs) by sparsifying vision-language interactions rather than reducing visual tokens, avoiding the information bottleneck created by token compression methods. A universal model is trained across multiple computational budgets, and a lightweight routing policy dynamically allocates visual computation per sample at inference time. VISOR achieves up to 8.6× FLOP savings while matching or exceeding state-of-the-art accuracy, with particular gains on fine-grained, high-resolution tasks where token reduction methods degrade.

Problem

Existing efficiency methods for LVLMs rely on visual token reduction (pruning or compression), which creates an information bottleneck that causes significant accuracy drops on complex, high-resolution tasks requiring fine-grained reasoning (e.g., DocVQA, ChartQA, OCRBench). No prior work cleanly separates the cost of maintaining visual information from the cost of cross-modal computation within the LLM layers.

Method

VISOR decouples text and visual token processing inside the LLM by replacing most full self-attention layers (which process concatenated \([V; T]\) sequences at cost \(O((N_v + N_t)^2 d)\)) with two sparse interaction types:

1. Cross-attention layers (\(l \in L_{CA}\), cost \(O(N_t N_v d)\)): Text tokens query static visual tokens; visual tokens \(V^{(l)}\) are never updated. This suffices for coarse tasks.
2. Selective self-attention layers (\(l \in L_{SA}\)): Full transformer layers applied to \([V; T]\), refining visual representations hierarchically and enabling subsequent cross-attention layers to access higher-level features. The update rule is: $$ (V^{(l)}, T^{(l)}) = \begin{cases} \text{TL}l([V^{(l-1)}; T^{(l-1)}]) & l \in L} \ (V^{(l-1)}, \text{TLl(\text{CrossAttn}(T^{(l-1)}, V^{(l-1)}))) & l \in L $$} \ (V^{(l-1)}, \text{TL}_l(T^{(l-1)})) & \text{otherwise} \end{cases

The maximum configuration uses \(|L_{CA}| = |L_{SA}| = L/3\). A universal model is trained by randomly sampling viable sub-configurations (varying the subset of \(L_{SA}\) layers) at each optimization step. An adaptive routing policy (MLP on a special routing token, trained via offline pseudo-labeling with 99%-accuracy threshold) dynamically selects the self-attention configuration per sample at inference time. Conditional positional embeddings are implemented via 1D depthwise convolution (kernel size 7) to preserve spatial structure. VISOR can additionally be composed with token reduction methods (VISOR-TR) for further FLOP savings.

Key Contributions

• Empirical analysis of cross-modal attention patterns and CKA-based visual feature evolution in LLaVA-OV, motivating sparse, layer-selective interaction design.
• Cross-attention-only VISOR variant that outperforms prior token reduction state-of-the-art on coarse tasks with no visual token compression.
• Selective self-attention insertion for progressive visual feature refinement, recovering fine-grained task performance lost by cross-attention-only approaches.
• Universal model training strategy enabling a single network to operate robustly across a range of computational budgets.
• Lightweight offline pseudo-label routing policy for per-sample adaptive computation allocation.
• Demonstration that VISOR is orthogonal to and composable with token reduction methods, reaching up to 35× FLOP savings.

Results

• Up to 8.6× FLOP savings over LLaVA-OV baseline on coarse tasks while matching or exceeding accuracy of token reduction baselines (VisionZip, HiRED, M³).
• On fine-grained ("hard") tasks (DocVQA, ChartQA, InfoVQA, OCRBench), VISOR outperforms all token reduction baselines, which suffer accuracy collapse due to information bottleneck.
• Combined VISOR-TR achieves up to 18× FLOP savings while maintaining state-of-the-art accuracy.
• Combining with more aggressive token reduction (4×+) yields up to 35× FLOP savings with only minor accuracy drop on hard tasks.
• Ablation: cross-attention alone saturates at ~8 CA layers for easy tasks; adding 7 SA layers nearly matches the full dense model on hard tasks.
• All comparisons use a unified LLaVA-OV (0.5B) backbone; additional results with 1.5B reported in supplementary material.

Limitations

• Evaluated primarily on LLaVA-OV; generalization to other LVLM architectures (e.g., Flamingo-style, decoder-only with no explicit cross-attention) is not demonstrated.
• The pseudo-labeling routing policy requires running the full universal model on a training subset offline, adding a labeling cost before deployment.
• Aggressive token reduction combined with VISOR still degrades on hard tasks, indicating the information bottleneck is not fully eliminated in the VISOR-TR regime.
• Maximum configuration search and viable sub-network identification involve systematic evaluation of layer subsets, which can be expensive for larger models.
• Training used a smaller subset of LLaVA-OV data due to data availability, potentially affecting absolute performance comparisons.

Relevance to Vision-Language Models

VISOR directly addresses the core efficiency-accuracy tradeoff in LVLMs by reframing the problem from "how many tokens to keep" to "how many interaction layers to execute," a fundamentally different axis of sparsification. This is highly relevant for the VLM community because it preserves the full high-resolution visual context—a key requirement for document understanding, chart reasoning, and dense OCR tasks—while achieving competitive or superior FLOP savings to token compression methods. The task-dependent analysis of attention patterns and CKA feature evolution provides a principled empirical foundation for future work on dynamic computation in multimodal transformers. The composability with token reduction methods suggests VISOR defines an orthogonal design dimension that can be layered onto virtually any existing LVLM efficiency technique.