Skip to content

Searching for Task-Specific Vision Paths: Evolutionary Block Pruning Across Vision-Language Models

🕒 Published (v1): 2026-07-19 03:43 UTC · Source: Arxiv · link

Why this paper was selected

Evolutionary block pruning finds task-specific vision paths; cuts VLM compute without retraining

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper asks whether fixed subsets of vision-encoder blocks can be skipped in frozen VLMs without fine-tuning, using a source-balanced evolutionary search to find these "routes." Evolutionary search reliably beats naive (independent-ranking, contiguous, random) block selection, but capability-specific routing is unstable across architectures and datasets.

Problem

Every VLM query runs the full vision encoder regardless of the visual capability required (OCR, counting, spatial, etc.). Prior training-free pruning work (e.g., Short-LVLM) searches for a single globally compressed model. It remains unknown (a) whether combinatorial block interaction invalidates independent-ranking pruning and (b) whether different visual capabilities benefit from different fixed block subsets.

Method

Identity block skipping. A route \(S \subseteq \{0,\ldots,L-1\}\), \(|S|=K\), replaces selected transformer blocks with identity operations: $\(h_{l+1} = \begin{cases} h_l, & l \in S \\ F_l(h_l), & l \notin S \end{cases}\)$ No parameters are removed and no fine-tuning is performed.

Source-balanced objectives. Routes are scored not by raw accuracy but by capability-source cell accuracy \(A^S_{c,d}\), with the objective vector \(q_\text{shared}(S) = (\mu, w, \sigma)\) — mean damage, worst-source damage, and source variability — to prevent a good mean from hiding severe per-source failures. For capability-specific routes, a collateral damage term \(\kappa_t\) discourages destroying other capabilities.

Evolutionary search. A seeded genetic algorithm with NSGA-II multi-objective Pareto survival runs for three generations. Initialization seeds up to half the population with prior single-block rankings. Crossover retains the intersection of two same-size parent routes and fills remaining slots from the non-shared union via SHA-256-ordered selection. Every offspring receives exactly one-swap mutation. Three fixed seeds (20260715–17) produce finalists evaluated on a 300-example dev set; the lowest-loss route on a 876-example image-disjoint selection set is frozen.

Controls. At matched budget \(K\): independent-ranking (lowest per-block accuracy drop), contiguous window, and three deterministic random routes.

Key Contributions

  • Source-balanced evolutionary search procedure for fixed-budget vision-block route construction, without fine-tuning.
  • Matched-budget empirical study across Qwen2.5-VL-3B-Instruct (32 vision blocks) and SmolVLM2-2.2B-Instruct (27 vision blocks) at \(K \in \{4, 6, 8\}\).
  • Formal analysis of pairwise block interaction \(I(i,j) = \Delta(\{i,j\}) - \Delta(\{i\}) - \Delta(\{j\})\) motivating combinatorial search.
  • Capability-level, cross-model, and source-transfer analysis separating search quality from capability-specificity claims.

Results

  • Evolutionary search vs. independent ranking (Qwen shared route): +1.03 pp at \(K=4\), +1.03 pp at \(K=6\), +3.54 pp at \(K=8\); at \(K=8\) contiguous removal collapses to 40.98% and random mean to 46.96% vs. 75.91% searched.
  • SmolVLM2 replication (\(K=4\)): searched shared route 72.49% vs. 67.58% independent ranking, +4.91 pp [1.83, 7.99].
  • Capability-specific routing (Qwen, \(K=6\)): policy 81.28% vs. shared 79.11%, +2.17 pp [0.00, 4.34] (borderline); OCR gain +7.10 pp [0.65, 14.19].
  • SmolVLM2 capability policy (\(K=4\)): only +0.80 pp overall [-1.94, 3.54]; counting +7.18 pp, spatial +9.39 pp, OCR −13.55 pp [-21.94, -5.16].
  • IIIT5K source-transfer (SmolVLM2, sealed): full model 94.8%; shared route 86.4%; OCR-specific route 72.8% — 13.6 pp below the shared route [-19.2, -8.4].

Limitations

  • Capability-specific routing is not stable across architectures: what gains OCR accuracy on Qwen damages it on SmolVLM2.
  • OCR-specific route on SmolVLM2 fails to transfer to a new OCR source (IIIT5K), ruling out the hypothesis that routes encode stable capability pathways.
  • Capability labels are assumed known at inference (oracle setting); no learned router is studied.
  • Search runs only three generations with populations of 12–16 and cannot guarantee global optima; finite search budget is a confound in negative capability results.
  • Only \(K \in \{4,6,8\}\) evaluated for Qwen; SmolVLM2 only at \(K=4\).
  • The 876-example selection set is not fully sealed (single-block discovery touched its sources), limiting strength of evidence claims.
  • Latency improvement is modest: a four-block skip yields only ~4.89% vision-encoder latency reduction.

Relevance to Vision-Language Models

This work directly targets the vision encoder of modern VLMs, demonstrating that block-level redundancy is real but non-compositional — single-block importance scores cannot predict safe multi-block routes, a finding that challenges the standard heuristic behind pruning methods like ShortGPT applied to vision towers. The negative capability-routing result is particularly significant for the VLM field: despite intuitive appeal, visual capabilities (OCR, spatial, counting) do not map cleanly to fixed blocks or depth ranges within the vision encoder, and this fails to generalize across architectures. For researchers studying efficient VLMs, the paper establishes evolutionary combinatorial search as a reliable training-free method for vision-encoder compression while issuing a caution against over-interpreting capability-to-block associations.