Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deployment¶
🕒 Published (v1): 2026-07-09 01:13 UTC · Source: Arxiv · Venue: ICML 2026 · link
Why this paper was selected
ICML 2026; component-wise quantization analysis for sub-3B VLMs — direct edge deployment recipe
Ask a follow-up
Open an assistant pre-loaded with this paper's context.
💬 Ask ChatGPT✦ Ask Claude
TL;DR¶
This paper presents a hypothesis-driven, component-wise evaluation framework for quantizing small VLMs (≤3B parameters) on heterogeneous edge SoCs (NVIDIA Jetson Orin NX/AGX). By independently ablating the vision encoder, projector, and LLM backbone across six quantization configurations on five architecturally diverse models, the authors derive five empirically validated deployment guidelines. Key findings overturn scale-centric intuitions and reveal hardware-software stack interactions invisible to server-side benchmarking.
Problem¶
Existing VLM quantization research evaluates models end-to-end on server-grade GPUs, leaving practitioners without principled guidance on which components to quantize, at what bit-width, and on what hardware. The heterogeneous nature of VLM components—vision encoder, cross-modal projector, and LLM backbone—means uniform quantization policies are suboptimal, yet controlled marginal-contribution studies and cross-platform edge comparisons are nearly absent.
Method¶
Five sVLMs (Qwen3-VL-2B, DeepSeek-VL2-Tiny, PaliGemma2-3B, LLaVA-OV-0.5B, Kosmos-2.5) are evaluated under six quantization configurations (Table 2):
| Config | Vision | Projector | LLM |
|---|---|---|---|
| cfg0 | FP16 | FP16 | FP16 |
| cfg1 | FP16 | FP16 | INT4 |
| cfg2 | FP16 | INT8 | FP16 |
| cfg3 | FP16 | INT8 | INT4 |
| cfg4 | INT8 | FP16 | FP16 |
| cfg5 | INT8 | FP16 | INT4 |
Quantization uses BitsAndBytes INT4/INT8; vision components are capped at INT8 because they represent the modality-alignment bottleneck. Evaluation axes are MME benchmark accuracy (max 2800), peak VRAM (GB), component-wise latency (\(L_V\), \(L_P\), TPOT in ms), and intelligence-per-joule (IPJ). Latency is decomposed by CUDA-synchronized timing around each component (Algorithm 1); VRAM is measured as peak allocation. All data points reflect \(n \geq 3\) independent runs on Jetson Orin NX (16 GB, 102.4 GB/s, 25W) and AGX (64 GB, 204.8 GB/s, 50W).
Key Contributions¶
- First controlled ablation experiment that isolates each of three VLM components (vision encoder, projector, LLM backbone) independently for sVLMs under edge SoC constraints.
- Empirical falsification of scale-dependent sensitivity: MoE vs. dense architecture dominates INT4 sensitivity, not parameter count alone; revised hypothesis H1\(_\text{rev}\) introduced.
- Discovery and mechanistic attribution of a SigLIP-So400m + BitsAndBytes INT8 + Jetson Ampere latency anomaly (2.43–4.66× encoder slowdown), decoupled from accuracy impact.
- Demonstration that LLM INT4 via BitsAndBytes should be treated as a VRAM-saving tool only, not a latency optimizer, on Ampere-class edge hardware.
- Characterization of non-additive composite quantization errors along the modality-alignment path (architecture-dependent super- and sub-additivity).
- Practical deployment guidelines: modality- and hardware-aware mixed-precision allocation strategies for edge environments.
Results¶
LLM INT4 accuracy (cfg1, \(\Delta\)MME vs. baseline): - Qwen3-VL-2B (MoE): +56.04 (NX), +101.88 (AGX) - DeepSeek-VL2-Tiny (MoE): +39.50 (NX), +48.18 (AGX) - PaliGemma2-3B (dense): −62.12 (NX), −59.35 (AGX) - LLaVA-OV-0.5B (dense): −220.02 (NX), −156.08 (AGX)
Vision INT8 encoder latency anomaly (cfg4 / cfg0 ratio): - SigLIP-So400m models: 2.43–4.66× slowdown on both platforms - SigLIP-2 (Qwen3-VL): 1.73–2.51× slowdown - Pix2Struct / Kosmos-2.5: 1.15–1.20× slowdown (minimal overhead)
LLM INT4 VRAM and TPOT trade-off (cfg1): - VRAM reduction: 21.8–47.5% across all models and platforms - TPOT increase: 10.4–56.3% across all models and platforms - Energy cost: Qwen3-VL INT4 increases energy consumption by 45.3–54.7%; LLaVA-OV INT4 by only 1.4–1.9% (due to small 0.49B LLM)
Composite quantization (cfg3, cfg5): - Most component interactions are approximately additive - PaliGemma2 cfg5 shows sub-additive residual degradation of −16.33 MME (NX) - DeepSeek-VL2 cfg5 shows super-additive pattern of −8.03 MME (NX)
Accuracy rankings are platform-invariant; latency/energy profiles are platform-specific (H5 confirmed).
Limitations¶
- Quantization restricted to BitsAndBytes INT4/INT8; AWQ, GPTQ, and SmoothQuant are not evaluated, so findings may not generalize to other quantization stacks.
- Jetson Orin Nano excluded due to OOM failures during weight conversion, limiting coverage of the lowest-resource tier.
- Only five model architectures tested; conclusions about MoE resilience rely on two MoE models.
- MME is the sole accuracy benchmark; task-specific or fine-grained multimodal benchmarks are not reported.
- Projector sensitivity (cfg2) is consistently near-zero across all models, but this may reflect MME's insensitivity rather than true projector robustness.
- The SigLIP-BitsAndBytes-Ampere latency anomaly is diagnosed at the kernel-interaction level but not patched; no alternative quantization kernel is proposed.
- INT4 inference on Ampere lacks native hardware support, making the TPOT findings specific to BitsAndBytes dequantization overhead on this architecture.
Relevance to Vision-Language Models¶
This work directly addresses a practical gap in VLM deployment: how to allocate quantization precision across heterogeneous components when targeting edge SoCs. The finding that MoE LLM backbones are substantially more resilient to INT4 quantization than dense counterparts is architecturally significant and should inform backbone selection for on-device VLMs. The SigLIP encoder latency anomaly—a hardware-software interaction invisible in server-side evaluation—demonstrates that encoder architecture choice has non-trivial downstream deployment consequences beyond accuracy. For researchers tracking VLMs, this paper provides the first component-level empirical substrate for mixed-precision design rules, complementing accuracy-focused leaderboard benchmarking with hardware-grounded efficiency analysis.