Skip to content

MVPruner: Dynamic Token Pruning for Accelerating Multi-view Vision-Language Models in Autonomous Driving

🕒 Published (v1): 2026-06-26 02:33 UTC · Source: Arxiv · link

Why this paper was selected

Dynamic token pruning for multi-view VLMs; practical efficiency for autonomous driving

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MVPruner is a two-stage adaptive token pruning framework for multi-view VLMs in autonomous driving. It dynamically allocates pruning budgets per camera view based on intra-view feature diversity in shallow layers and instruction-text relevance in deeper layers. On DriveLM with 90% token reduction, it retains 98.5% accuracy with a 4.97× prefilling speedup.

Problem

Existing visual token pruning methods apply uniform pruning ratios across all camera views, ignoring the heterogeneous importance of different views (e.g., front camera is critical at a red light; rear cameras are not). Furthermore, static importance metrics and single-stage pruning ignore how a VLM's attention pattern shifts layer-by-layer from global semantic diversity to local task-relevant focus, leading to suboptimal token retention.

Method

MVPruner applies two pruning stages at fixed layer boundaries (before layers 0 and 16 in the tested models):

Stage 1 — Diversity-aware Ratio Allocation (DRA) + Cross-stage Contribution-aware Token Selection (CCTS): Each view \(p\) receives a retention count proportional to its normalized diversity score \(S_p^u\), computed as the mean per-token uniqueness \(S_{p,i}^u = \min_j d_p(X_{p,i}^v, X_{p,j}^v)\) where \(d_p\) is cosine distance. Token selection uses a greedy iterative scheme: the token with highest cosine similarity to instruction tokens initializes the set, then each subsequent token is chosen to maximize \(\text{Imp}_{p,i} = U_{p,i} \times R_{p,i}\) (uniqueness × task relevance), balancing immediate visual diversity with future task utility.

Stage 2 — Instruction-aware Ratio Allocation (IRA) + instruction-guided selection: In deeper layers where cross-modal attention has localized, each view's budget is reallocated using \(r_{2,p} = r_2 \times (1 + S_p^r - S_p^u)\), where \(S_p^r\) is the mean cross-attention score between the view's remaining tokens and the instruction tokens. This compensatory normalization boosts views that are task-relevant but not necessarily diverse. Final token selection retains the top-\(k_{2,p}\) tokens by their attention scores to instruction tokens.

Both allocation strategies operate entirely online (no offline hyperparameter search).

Key Contributions

  • Empirical analysis showing multi-view VLMs encode task-related view priors in deeper layers and exhibit a diversity-driven → task-relevance-driven attention shift across layers.
  • DRA: per-view pruning budget proportional to intra-view feature diversity for the shallow stage.
  • CCTS: greedy token selection jointly scoring semantic uniqueness and cross-modal task relevance to maximize cross-stage utility.
  • IRA: per-view budget reallocation in deep layers guided by text-to-vision attention scores, with compensatory normalization relative to diversity scores.
  • Demonstrated on both image-based (DriveLM, DriveLMM-o1, MAPLM) and video-based (STSnu) benchmarks without offline tuning.

Results

  • DriveLM (90% pruning, 10% tokens retained): 98.5% accuracy retained vs. original DriveMM; 4.97× prefilling speedup; 87.3% FLOPs reduction. Outperforms second-best Prune2Drive by +0.8 on average score.
  • DriveLMM-o1 (90% pruning): 98.0% accuracy retained; outperforms Prune2Drive by +2.6 on Overall Reasoning (72.91 vs. 70.31).
  • MAPLM (90% pruning): average score drops only 3.94 points vs. baseline; FastV drops 17.63, SparseVLM 15.84, Prune2Drive 7.43.
  • STSnu (75% pruning, 90% pruning): 100% and 97.3% of baseline retained, respectively; at 75% pruning, MVPruner matches the uncompressed model exactly.
  • DriveLM (75% pruning): surpasses the uncompressed baseline on ChatGPT score, Rouge, CIDEr, and Match metrics.

Limitations

  • Stage boundary (layer 0 and layer 16) is fixed and not automatically tuned per model architecture; different backbone depths may shift the diversity-to-relevance transition point.
  • CCTS greedy selection is \(O(K \cdot M)\) per view per stage; with large \(M\) and many views this adds non-trivial overhead not fully quantified in the paper.
  • Evaluated only on two specialist driving VLMs (DriveMM, DriveLMM-o1); generalizability to general-purpose multi-view VLMs is unverified.
  • The compensatory normalization in IRA (Eq. 6) mixes diversity and relevance scores on different scales without explicit normalization, which may behave unpredictably when \(S_p^r \ll S_p^u\).
  • No ablation on the choice of instruction-token subset (system text vs. query text) used for relevance scoring.

Relevance to Vision-Language Models

MVPruner addresses a core efficiency bottleneck for deploying VLMs in latency-constrained settings: the quadratic cost of long multi-image token sequences in the prefill phase. The mechanistic finding — that VLM cross-modal attention transitions from global diversity-driven to local relevance-driven across layers — is a generalizable observation applicable beyond driving, informing how any multi-image VLM should schedule compression. The work complements token merging and attention-head pruning literature by showing that view-adaptive, two-stage pruning grounded in model-internal attention signals substantially outperforms uniform or statically tuned approaches, and that near-lossless compression (\(>\)98% accuracy at 10% token budget) is achievable without fine-tuning.