Skip to content

LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; Fourier approximation compresses large multimodal models with strong efficiency

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LLaVA-FA compresses large multimodal models by performing joint low-rank plus quantization decomposition in the Fourier (frequency) domain rather than the spatial domain. The key insight is that 2D-DFT de-correlates weight matrices, yielding more compact singular value spectra and conjugate symmetry that halves the number of learnable parameters. The result is a family of efficient VLMs (1B–7B activated parameters) that outperform same-scale baselines across comprehension and hallucination benchmarks.

Problem

Existing LMM compression methods decouple low-rank decomposition from quantization, compounding reconstruction errors. Worse, multimodal architectures carry cross-modal adapters (vision encoder, VLP projector) whose ranks balloon beyond what pure-text LLM recipes handle, leaving the multimodal stack disproportionately large even after standard low-rank + quantization treatment.

Method

For each weight matrix \(W \in \mathbb{R}^{d_1 \times d_2}\), LLaVA-FA applies 2D-DFT to obtain \(\tilde{W} \in \mathbb{C}^{d_1 \times d_2/2}\) (conjugate symmetry halves the columns). It then solves:

\[\min_{\tilde{Q},\tilde{L}_1,\tilde{L}_2} \left\|\sqrt{C} \odot \left(\tilde{W} - (\tilde{Q} + \tilde{L}_1\tilde{L}_2)\right)\right\|_F\]

where \(\tilde{L}_1 \in \mathbb{C}^{d_1 \times r}\), \(\tilde{L}_2 \in \mathbb{C}^{r \times d_2/2}\) are the low-rank factors (from truncated complex SVD keeping top-\(r\) singular values) and \(\tilde{Q}\) is the quantized residual. The calibration weight \(C\) is set to element-wise squared activation expectations over a small held-out set.

PolarQuant quantizes the complex residual \(\tilde{Q}\) in polar coordinates: amplitude \(r_{ij}\) with \(b_r\) bits (uniform grid over \([0, r_{\max}]\)) and phase \(\theta_{ij}\) with \(b_\theta\) bits (uniform grid over \([-\pi, \pi]\)), then reconstructs as \(\hat{Q}_{ij} = \hat{r}_{ij} e^{i\hat{\theta}_{ij}}\).

Optional Diagonal Calibration (ODC) approximates the intractable calibration-weighted SVD by replacing the full calibration matrix with diagonal row/column mean matrices \(D_\text{row}\), \(D_\text{col}\), solving \(\min \|D_\text{row} E D_\text{col}\|_F\) via FourierSVD — eliminating the need for large calibration datasets. The overall algorithm alternates between ODC and PolarQuant until error \(\epsilon_t > \epsilon_{t-1}\).

LLaVA-FA applies this scheme uniformly to the CLIP-ViT-L/14 vision encoder, the two-layer MLP projector, and the LLM backbone (Qwen-2.5, InternLM-2, or LLaMA-3 depending on scale). Compressed models are trained on 5M samples on 8× RTX 4090 GPUs.

Key Contributions

  • Frequency-domain joint compression: first method to perform low-rank + quantization approximation jointly in the Fourier domain for LMMs, exploiting de-correlation and conjugate symmetry.
  • Theoretical guarantee: proves (Lemma 3.1) that frequency-domain truncation yields smaller Frobenius error than spatial-domain truncation at the same rank, because the frequency domain produces more compact singular value spread.
  • PolarQuant: a polar-coordinate complex-matrix quantizer that separately encodes amplitude (\(b_r\) bits) and phase (\(b_\theta\) bits), preserving cross-modal phase structure.
  • Optional Diagonal Calibration (ODC): approximates the full Hessian with row/column means, enabling calibration-free (or minimal-data) deployment.
  • Bit-budget analysis: derives \(B_\text{avg}\) formula (Eq. 3) showing the compression scheme yields sub-full-precision bits per parameter for standard LLM layer dimensions.

Results

  • LLaVA-FA-7B (compressed from InternLM-2-20B, 5M samples): AVG 74.5 across GQA/VisWiz/SQA\(^\text{I}\)/TextVQA/MME/MMB/MMB-CN, outperforming LLaVA-1.5-7B (AVG 70.4), LLaVA-NeXT (73.6), Deepseek-VL-7B (68.5), and Qwen-VL-Chat (64.4).
  • LLaVA-FA-3B (from LLaMA-3-8B, 5M samples): AVG 70.5, beating MiniCPM-V-2 (68.5), MoE-LLaVA-3B (65.2), VILA-3B (63.4).
  • LLaVA-FA-2B (from Qwen-2.5-7B, 5M samples): AVG 60.9 on 7-benchmark suite; GQA 62.1, TextVQA 59.3 — best in the ∼2B class, outperforming DeepSeek-VL-1.3B (58.5) and MoE-LLaVA-2B (55.3).
  • LLaVA-FA-1B (from Qwen-2.5-3B, 5M samples): AVG 56.7 vs. SPHINX-Tiny 49.2; SQA\(^\text{I}\) 61.3 vs. 21.5.
  • Training cost: uses \(<0.25\%\) calibration data and \(<0.3\%\) trainable parameters relative to comparable LMMs, with notably fewer GPU hours than Qwen-VL-Chat or LLaVA-1.5-7B (see Fig. 1).

Limitations

  • Fourier approximation introduces complex arithmetic overhead at inference (IDFT must be applied before or fused into the forward pass); the paper does not report wall-clock inference latency, only parameter counts.
  • The conjugate symmetry saving applies only to real-valued weight matrices transformed by 2D-DFT; matrices that are already complex (e.g., certain positional encodings) do not benefit equally.
  • ODC accuracy depends on the empirical diagonal-dominance assumption of the Hessian; this approximation can degrade for layers with strongly off-diagonal curvature.
  • Experiments are confined to LLaVA-style architectures; generalizability to encoder-decoder or diffusion-based VLMs is untested.
  • Table 1 does not report actual memory footprint (GB) or FLOPs, making it difficult to directly compare hardware efficiency with token-pruning methods like LLaVA-PruMerge.

Relevance to Vision-Language Models

LLaVA-FA addresses a fundamental bottleneck for deploying VLMs on resource-constrained hardware: the multimodal stack (vision encoder + projector + LLM) is harder to compress than a pure-text LLM because cross-modal adapters resist the standard low-rank treatment. By reformulating compression entirely in the Fourier domain, this work provides a principled alternative to token pruning (CrossGET, LLaVA-PruMerge) and weight pruning (UPop), achieving drop-in weight replacement without architectural changes. The PolarQuant codec is directly applicable to any VLM component that produces complex-valued frequency representations after DFT. For researchers tracking efficient VLMs, this paper establishes frequency-domain joint compression as a competitive paradigm and provides Lemma 3.1 as a theoretical anchor for why the frequency domain is preferable for low-rank approximation of weight matrices.