Skip to content

Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference

🕒 Published (v1): 2026-06-30 16:08 UTC · Source: Arxiv · link

Why this paper was selected

Operator-level visual token skipping; directly cuts MLLM inference cost

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Decomposes each Transformer layer in MLLMs into attention and FFN operators, then selectively bypasses redundant operators while preserving the full visual-token sequence. Introduces answer-observable diagnostics showing that late-layer visual-token updates can be large yet "answer-silent," and uses operator-risk analysis to assign per-layer skipping policies. Achieves 33.7% FLOP reduction on Qwen3-VL with 99.5% of vanilla performance across 10 VQA benchmarks.

Problem

Existing MLLM inference acceleration methods operate at coarse granularities: token pruning irreversibly discards visual evidence and disrupts spatial structure; layer-wise skipping treats attention and FFN as an inseparable block, suppressing useful operators alongside redundant ones. Neither can decide whether attention, FFN, or both are redundant at each layer, and methods like V-Skip assume a fixed preference for preserving FFN while skipping visual attention.

Method

  1. Answer-observable diagnostics: Define Visual Update Magnitude (VUM), Answer-Observable Influence (AOI, via Jacobian-vector product from visual states to final-prompt-token hidden state), and Answer-Observable Efficiency (AOE, AOI normalized by VUM). Find that late layers exhibit rising VUM but falling AOI/AOE—visual states change substantially yet have little effect on the answer.

  2. Operator risk analysis: For each layer, measure KL divergence between full-model output and output after separately ablating visual attention (\(R_A\)), visual FFN (\(R_F\)), or both (\(R_{AF}\)). Define dominance ratio \(\Gamma_l = \log\frac{R_A(l)+\epsilon}{R_F(l)+\epsilon}\): positive → attention-dominant, negative → FFN-dominant. Layers with uniformly small \(R_{AF}\) are "frozen."

  3. Operator-aware skipping policy: Given a compute budget \(B\), rank layers by \(R_A(l)+R_F(l)\) and select the \(B\) lowest-risk layers. Assign per-layer policy: FFN-ONLY (skip visual attention) when \(R_A < R_F\), ATTENTION-ONLY (skip visual FFN) when \(R_F < R_A\), FREEZE-UPDATE (skip both) when \(R_{AF}\) is small. Textual tokens follow full computation; full visual sequence is retained.

  4. FLOP reduction: Skipping visual attention saves \(\Theta(N_v d^2 + N_v d)\), skipping visual FFN saves \(\Theta(\kappa N_v d d_{ff})\). The policy is fixed after a single calibration pass (7 samples from each of 6 benchmarks).

Key Contributions

  • Identifies answer-silent visual-token updates: Late-layer visual states can change substantially while having near-zero influence on answer-token representations, exposing a finer form of redundancy missed by update-magnitude metrics.
  • Reveals operator-level dominance: Useful visual computation in MLLMs is operator-dominant and layer-dependent—some layers are attention-dominant, others FFN-dominant, others redundant for both.
  • Operator-level visual skipping framework: First method to selectively skip attention, FFN, or both per layer while preserving the full visual-token sequence, avoiding both irreversible evidence loss and block-level over-pruning.
  • Strong efficiency-accuracy trade-off across architectures: Consistent gains on LLaVA-1.5-7B, Qwen2.5-VL-7B, and Qwen3-VL-8B across 10 VQA benchmarks; complementary with token pruning methods.

Results

  • Qwen3-VL-8B: 33.7% TFLOP reduction with 99.5% average performance retention (vanilla baseline). Outperforms VTW (75.8%), ShortV (97.1%), V-Skip (96.3%), V2Drop (96.0%), and APET (98.0%) on Avg. Retention.
  • LLaVA-1.5-7B: 34% TFLOP reduction at 98.8% Avg. Retention; second-best behind V-Skip+ (99.6%, which uses dataset-specific optimal settings).
  • Qwen2.5-VL-7B: 45% TFLOP reduction at 95.0% Avg. Retention, highest among all methods at comparable compute (V2Drop 93.8%, APET 94.5%).
  • Budget sensitivity: Performance degrades gracefully. On Qwen3-VL, budget 20 → 99.5% Avg. Ret.; budget 36 → 87.7% Avg. Ret.
  • Ablation: Operator-aware assignment outperforms fixed rules (ATTN-SKIP, FFN-SKIP, FROZEN) at the same budget on GQA, POPE, MME, OCRB, VizWiz.
  • Complementarity with token pruning: Applying operator skipping on top of VisionZip (10% tokens) recovers performance to near-full while further reducing FLOPs (1.31 TFLOPs vs 1.67 TFLOPs for VisionZip alone).
  • First-token latency: On TextVQA with LLaVA-1.5-7B, 49.3 ms prefill (vs ShortV 45.0 ms) and 110.4 ms total latency (vs vanilla 120.4 ms).

Limitations

  • Model-dependent budget: Optimal skipping budget varies by architecture (20/36 layers for Qwen3-VL vs 20/28 for Qwen2.5-VL) and must be tuned per model as an efficiency–performance knob.
  • Requires internal access: Risk estimation needs per-layer hidden states and Jacobian-vector products; not applicable to black-box API models.
  • Wall-clock speedup is implementation-dependent: Theoretical FLOP reduction does not directly translate to latency gains without kernel-level optimizations (e.g., attention masking, sparse FFN).
  • Calibration set overhead: A small calibration pass across 6 benchmarks (42 samples total) is needed per model; policy is then fixed, but this still adds a pre-deployment step.
  • Aggressive budgets degrade fine-grained tasks: Performance on OCR-heavy and text-rich benchmarks (OCRB, TextVQA, VizWiz) drops fastest under high skipping budgets.

Relevance to Vision-Language Models

Directly addresses the growing inference bottleneck caused by high-resolution inputs and dense visual encoders expanding visual token sequences to thousands of tokens. The finding that useful visual computation is operator-dominant (not block-aligned) reframes how the VLM community should think about redundancy in multimodal backbones: the default assumption that attention always matters more than FFN (or vice versa) is false. The method is complementary to token pruning, suggesting a two-tier redundancy hierarchy—spatial (which tokens) and functional (which operators)—that future efficient VLM designs can exploit.