Skip to content

SAB-LVLM: Significance-Aware Binarization for Large Vision-Language Models

🕒 Published (v1): 2026-07-02 08:30 UTC · Source: Arxiv · link

Why this paper was selected

Binarization compression for large VLMs; practical inference efficiency for deployment

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SAB-LVLM is a post-training binarization method for Large Vision-Language Models that compresses weights to ~1 bit while respecting the heterogeneous importance of weights across layers and modalities. It constructs per-modality Hessian-based sensitivity maps, combines them into a significance-aware binarization map \(\Lambda\), and uses \(\Lambda\) as an error-reweighting term in the quantization objective, then refines binary parameters via alternating significance-weighted updates.

Problem

Existing PTQ binarization methods (PB-LLM, BiLLM, ARB-LLM) minimize global weight quantization error \(\|\mathbf{W} - \hat{\mathbf{W}}\|_F^2\) uniformly, ignoring the heterogeneous contribution of weights across layers and modalities in LVLMs. This causes task-irrelevant unimodal encoding parameters to be over-preserved while modality-critical cross-modal weights are under-optimized, producing severe accuracy degradation under ~1-bit compression.

Method

Step 1 — Per-modality Hessian construction. \(K=128\) paired calibration samples \(\{x_k^t, x_k^v\}\) from COCO 2017 are fed separately through the LVLM. Hidden states \(E_k^t, E_k^v\) yield Hessian matrices \(\mathbf{H}^t = \sum_k 2(E_k^t)^\top E_k^t + \eta^t \mathbf{I}\) and \(\mathbf{H}^v\) analogously, with diagonal damping \(\eta = \text{Tr}(\sum_k H_k)/m\).

Step 2 — Sensitivity matrix. For each weight \((i,j)\), \(\mathcal{S}_{ij} = \text{Concat}(\mathcal{S}_{ij}^t, \mathcal{S}_{ij}^v)\) where \(\mathcal{S}_{ij}^t = W_{ij}^2 / (H_{jj}^t)^{-2}\) measures how much perturbing that weight hurts the textual (resp. visual) path.

Step 3 — Spatial significance map \(\Gamma\). Using the Hoyer sparsity measure over \(\mathcal{S}_{ij}\): $\(\Gamma_{ij} = \frac{\sqrt{z} - \|\mathcal{S}_{ij}\|_1/\|\mathcal{S}_{ij}\|_2}{\sqrt{z}-1} \in (0,1)\)$ \(\Gamma_{ij} \to 1\) when the weight is activated by a single modality (sparse sensitivity); \(\Gamma_{ij} \to 0\) when both modalities activate it equally (dense sensitivity).

Step 4 — Modality-guided integration. A scalar modality integration score \(r \in (0,1)\) measures the fraction of total sensitivity residing in unimodal weights (those with \(\Gamma_{ij} > \tau\)). The final significance-aware map is: $\(\boldsymbol{\Lambda} = \underbrace{r \cdot \boldsymbol{\Gamma}}_{\text{unimodal}} + \underbrace{(1-r)(1-\boldsymbol{\Gamma})}_{\text{multimodal}}\)$

Step 5 — Weighted binarization objective. \(\Lambda\) replaces the flat Frobenius penalty: $\(\mathcal{L}(\mathbf{W}, \hat{\mathbf{W}}) = \|\sqrt{\boldsymbol{\Lambda}} \odot (\mathbf{W} - \hat{\mathbf{W}})\|_F^2\)$ Theorem 1 proves \(e_{ij}^* \propto \Lambda_{ij}^{-1}\), so high-significance weights incur smaller quantization error at optimality.

Step 6 — Alternating significance-weighted update. Weights are represented via a \(T\)-order binary expansion \(\hat{\mathbf{W}} = \sum_{\epsilon=1}^T (\alpha_r^\epsilon (\alpha_c^\epsilon)^\top) \odot \mathbf{B}^\epsilon\) (\(T=2\) for key weights, \(T=1\) otherwise). Row- and column-wise scaling vectors \(\alpha_r^\epsilon, \alpha_c^\epsilon\) are updated in closed form by setting \(\partial\mathcal{L}/\partial\alpha_r^\epsilon = 0\) and \(\partial\mathcal{L}/\partial\alpha_c^\epsilon = 0\), iterating for 15 rounds.

Key Contributions

  • First systematic exploration of weight binarization for LVLMs under a PTQ framework.
  • Spatial significance map \(\Gamma\) (Hoyer-based sparsity over per-modality Hessian sensitivities) that distinguishes unimodal from cross-modal weight activation.
  • Modality integration score \(r\) that adaptively blends unimodal and multimodal significance into \(\Lambda\), preventing the map from biasing purely toward single-modality information.
  • Alternating significance-weighted update scheme that incorporates \(\Lambda\) into closed-form scale updates for \(T\)-order binary expansion.
  • Theoretical guarantee (Theorem 1) that optimal quantization error is inversely proportional to significance weight.

Results

All baselines are at ~1.07–1.08 bit unless noted; memory measured on a single GPU.

Qwen2.5-VL-7B-Instruct (16-bit: 15.45 GB → SAB-LVLM: 2.14 GB): - MMStar: SAB-LVLM 45.79 vs. ARB-LLM 38.10, BiLLM 29.97, PB-LLM 4.6 - DocVQA: 85.34 vs. 80.48 / 33.08 / 9.40 - TextVQA: 74.00 vs. 70.70 / 53.38 / 28.67 - Video-MME: 48.11 vs. 47.14 / 26.56 / 9.89

Qwen2.5-VL-32B-Instruct (16-bit: 62.31 GB → SAB-LVLM: 10.28 GB): - MMStar: 54.77 vs. ARB-LLM 48.48; DocVQA: 90.36 vs. 90.00

Qwen2.5-VL-72B-Instruct (16-bit: 136.74 GB → SAB-LVLM: 23.23 GB): - MMStar: 58.48 vs. ARB-LLM 57.45; DocVQA: 92.56 vs. 90.42; Video-MME: 64.93 vs. 61.15

InternVL3.5-8B-Instruct (16-bit: 15.89 GB → SAB-LVLM: 2.28 GB): - MMStar: 27.02 vs. ARB-LLM 24.67; TextVQA: 42.04 vs. 30.83; VSI-Bench: 9.63 vs. 0.16

InternVL3.5-38B (16-bit: 71.51 GB → SAB-LVLM: 10.27 GB): - MMStar: 56.22 vs. ARB-LLM 54.54; DocVQA: 87.19 vs. 86.85; VSI-Bench: 43.00 vs. 41.62

SAB-LVLM consistently surpasses all binary PTQ baselines (PB-LLM, BiLLM, ARB-LLM) across both model families and all five benchmarks.

Limitations

  • Calibration data is fixed to COCO 2017 image–text pairs; out-of-distribution downstream tasks (e.g., specialized medical or chart VQA) may require domain-adapted calibration.
  • Only \(z=2\) modalities are considered; extension to audio or video-only modalities is not addressed.
  • Performance at ~1.07-bit still degrades substantially below 16-bit full precision, especially for smaller models (InternVL3.5-8B DocVQA: 48.03 vs. 92.47 full precision).
  • The method is PTQ-only with no fine-tuning, so the gap to quantization-aware training is not characterized.
  • Evaluation is limited to two LVLM families (Qwen2.5-VL and InternVL3.5); generalization to architectures with different cross-modal connectors (e.g., Q-Former-based) is untested.
  • Threshold \(\tau\) for the unimodality mask \(\mathcal{M}^{\text{uni}}\) requires tuning and its sensitivity is not ablated in the provided text.

Relevance to Vision-Language Models

This paper directly addresses the deployment bottleneck of LVLMs on edge devices by showing that the multi-modal architecture of VLMs demands modality-aware, not modality-agnostic, compression. The core insight — that cross-modal weights and unimodal weights warrant different quantization pressure — is broadly applicable to any VLM that fuses separate visual and textual encoders through a shared backbone. The work extends the PTQ binarization line from LLMs (BiLLM, ARB-LLM) into the VLM domain, establishing a new baseline for ~1-bit multimodal model compression and motivating future work on modality-specific quantization schedules, mixed-precision strategies, and calibration data selection for VLMs.