VLM in a flash: I/O-Efficient Sparsification of Vision-Language Model via Neuron Chunking¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Deploying large VLMs on flash-memory-constrained edge devices suffers from high I/O latency because conventional activation sparsification selects neurons by magnitude alone, producing fragmented, non-contiguous flash reads. Neuron Chunking couples neuron importance with flash access cost by selecting contiguous groups of neurons (chunks) that maximize importance per unit latency. On Jetson Orin hardware it reduces I/O latency by up to 5.76× at comparable accuracy versus top-k magnitude baselines.

Problem¶

Existing activation sparsification methods (e.g., TEAL, Deja Vu, CATS) are model-centric: they pick neurons ranked by activation magnitude and implicitly assume I/O latency scales linearly with data volume. This assumption breaks on flash storage, where latency depends strongly on access contiguity. VLMs exacerbate the problem because gated activations (SwiGLU/GeLU) and averaging over many visual tokens produce smooth, nearly uniform activation distributions, so top-k selection scatters reads across memory and severely degrades flash throughput.

Method¶

Neuron Chunking operates in two algorithmic components plus an offline preprocessing step:

Chunk-based latency model. Access patterns are abstracted into a contiguity distribution—the histogram of sizes of maximal contiguous groups (chunks) of selected neuron indices. Total I/O latency is approximated as the sum of per-chunk latencies from a one-time offline profiled lookup table T[s]. This avoids exhaustive profiling while remaining accurate (R² ≥ 0.83–0.99 across devices and models).
Utility-guided chunk selection. At runtime, sliding windows of multiple sizes generate candidate chunks over neuron indices. Each chunk receives a utility score = sum of neuron importance values / estimated chunk latency. A greedy algorithm sorts candidates by utility and iteratively selects the highest-scoring non-overlapping chunks until the channel budget R is met. Because the latency model error is approximately linear (proportional bias), relative utility rankings are preserved and the greedy selection remains correct.
Hot-cold offline reordering. Neurons are permuted offline in decreasing order of activation frequency (top-50% by importance counted on a calibration set), placing frequently activated neurons contiguously in the weight matrix. This preprocessing boosts baseline contiguity without co-activation statistics and contributes ~1.23× speedup; online chunk selection provides the dominant additional gain.

Key Contributions¶

Characterization of the mismatch between conventional activation sparsification and flash I/O access patterns in modern VLMs (smooth activation distributions → fragmented reads → increased latency at moderate sparsity).
Contiguity distribution as a compact, runtime-efficient abstraction of flash access behavior.
Chunk-based latency model: offline-profiled lookup table enabling per-matrix latency estimation in ~2 ms.
Utility-guided chunk selection: greedy algorithm that jointly maximizes neuron importance and minimizes flash I/O latency.
Empirical validation on Jetson Orin Nano and AGX across five VLMs (LLaVA-OneVision-7B/0.5B, VILA1.5-8B, NVILA-Lite-2B, LongVA-7B) and three benchmarks.

Results¶

Average I/O speedup on Jetson Orin Nano: 2.19× at comparable accuracy (linear interpolation); maximum 4.65×.
Average I/O speedup on Jetson Orin AGX: 2.89×; maximum 5.76×.
Larger gains on AGX reflect its wider throughput gap between contiguous and scattered access.
At 5% accuracy drop, end-to-end latency still decreases despite slight compute overhead from loading marginally more channels.
Ablation (LLaVA-OneVision-7B): hot-cold reordering alone yields ≤1.23×; adding chunk selection increases this to ≤2.55×.
Average chunk size increases from ~1–2 neurons (baseline) to ~50 neurons after reordering + chunking.
Chunk selection overhead: ~2 ms per weight matrix, ~400 ms for full model—small relative to total inference time.
Baselines: top-k magnitude sparsification with TEAL layer-wise sparsity profiles; same profiling budget applied to both.

Limitations¶

End-to-end speedup is smaller than I/O-only gain because compute time is constant; I/O–compute overlap (not applied) could narrow this gap.
Greedy chunk selection does not account for synergies among adjacent low-scoring chunks that could collectively form a more latency-efficient region.
Latency model accuracy degrades for smaller models or lower-end devices (R² as low as 0.832) due to lower I/O concurrency amplifying controller tail latency.
Hot-cold reordering alone provides modest gains; the method's full benefit requires online chunk selection, which cannot be precomputed offline.
If sufficient device memory is available to cache most model weights, flash I/O volume becomes negligible and the method's advantage diminishes.
Emerging async I/O mechanisms (io_uring) could reduce the performance gap between scattered and contiguous reads, partially eroding the method's advantage over time.

Relevance to Vision-Language Models¶

This paper directly targets a practical deployment bottleneck for VLMs on edge devices: the combination of large model size, multimodal weight loading, and flash I/O constraints creates a regime where standard sparsification techniques fail. The key VLM-specific insight—that gated activations (SwiGLU/GeLU) and multi-token visual averaging produce smoother activation distributions than ReLU LLMs, making storage-aware rather than pure magnitude-based selection critical—is broadly applicable to the current generation of VLM architectures. For researchers tracking VLMs, this work establishes hardware-aware structured sparsification as a necessary design consideration for real-time streaming VLM applications (AR, robotics), complementing orthogonal compression directions like quantization and pruning.