Skip to content

ScAle: Attention Head Scaling as a Minimal Adapter for Spatial Reasoning in Vision Language Models

🕒 Published (v1): 2026-06-28 19:48 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

Minimal efficient adapter for spatial reasoning, a known VLM weakness; Yanzhi Wang group

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ScAle learns a handful of bounded scalar coefficients (~1K params) that modulate last-token attention-head and MLP activations in a frozen VLM backbone, achieving up to 134.1% relative gains on SpatialEval without fine-tuning any pretrained weights. It recovers a substantial fraction of full LoRA performance while requiring 2,500Ă— fewer trainable parameters, and generalizes across LLaVA and Qwen2.5-VL families.

Problem

VLMs exhibit severe deficits in spatial reasoning (geometric relations, navigation, map understanding), scoring only ~20--30% on the SpatialEval benchmark. Standard approaches require fine-tuning millions of parameters (LoRA, full fine-tuning) or rely on hand-designed prompting, both of which are costly or insufficient. The paper hypothesizes that spatial reasoning failures arise from mis-calibrated competing modes in the LLM backbone: the 2-D spatial signal is encoded in a redundant 1-D token stream and obscured by interference from other activated modes.

Method

  • Bounded last-token scaling: For each attention head \(h\) in layer \(l\), a learnable scalar \(u^{\text{attn}}_{l,h}\) modulates the last-token output via \(\text{eff}(u) = 1 + s_{\max}\tanh(u)\) (clamped to \((1 - s_{\max}, 1 + s_{\max})\) with \(s_{\max}=10\)). The scaled activation \(\tilde{z}_{l,h} = \text{eff}(u^{\text{attn}}_{l,h}) \cdot z_{l,h}\) is then passed to \(W^O\).
  • MLP scaling (optional): one scalar per layer scaling the last-token MLP output: \(\tilde{Y}_l = \text{eff}(u^{\text{MLP}}_l) \cdot Y_l\).
  • Training: frozen backbone; only the \(L \times H + L\) scalars are optimized via cross-entropy with \(\ell_2\) regularization for 5 epochs on 5--20% of the training data (75--300 samples). Total parameter count: ~1K (e.g., 1,024 for LLaVA-7B: 32 layers \(\times\) 32 heads).
  • Variants: attention-only, MLP-only, or both. Attention scaling consistently dominates MLP scaling.

Key Contributions

  • Demonstrates that activation reweighting in a single frozen layer can substantially shift spatial reasoning accuracy (preliminary brute-force sweep).
  • Proposes ScAle, an ultra-lightweight adapter (~1K params) that learns bounded per-head scalars applied only to the last token, requiring no new modules or low-rank decompositions.
  • Achieves consistent gains across two model families (LLaVA-Next 7B/13B, Qwen2.5-VL 3B/7B) and three data regimes (5/10/20% training splits), with over three orders of magnitude parameter efficiency vs. LoRA.

Results

  • SpatialEval (LLaVA-7B, 20% training): Maze-Nav 31.7% \(\to\) 68.9% (+117.3% relative), Spatial-Grid 29.3% \(\to\) 68.6% (+134.1%), Spatial-Map 22.7% \(\to\) 45.0% (+98.2%) with only 1,024 attention-scaling parameters.
  • Parameter efficiency: ScAle (1K params) approaches LoRA-all (2.65M params) while using 2,500\(\times\) fewer; outperforms LoRA-last (78K params) by large margins despite being 76\(\times\) smaller; 600\(\times\) fewer params than (IA)\(^3\).
  • Cross-architecture: LLaVA-13B Maze-Nav 26.7% \(\to\) 69.8% (1.6K params); Qwen2.5-VL-3B Maze-Nav 26.0% \(\to\) 70.3% (576 params); Qwen2.5-VL-7B Maze-Nav 38.3% \(\to\) 66.2% (784 params).
  • Real-world VQA (WhatsUp-VLM): LLaVA-7B improves from ~56% to ~92%+ on VGQA/COCOQA; Qwen2.5-VL-3B from ~84% to ~94%.
  • Hallucination (POPE): LLaVA-7B 82.1% \(\to\) 88.2%, matching (IA)\(^3\) (89.8%) with 600\(\times\) fewer params. Qwen2.5-VL-3B 87.3% \(\to\) 90.4%, matching (IA)\(^3\) exactly.
  • Cross-task: Training on spatial tasks preserves non-spatial VQA accuracy; scaling learned on COCOQA/VGQA transfers reasonably well across QA tasks but less across spatial sub-tasks.

Limitations

  • MLP-only variant is ineffective (too few scalars), and attention scaling alone carries the results.
  • Still lags behind full LoRA-all on the hardest tasks (Spatial-Map gap remains large: 45.0% vs. 83.2%).
  • Requires task-specific training; no single universal set of scaling coefficients works across all tasks.
  • Evaluated primarily on spatial reasoning; generalizability to other reasoning modalities (temporal, causal, mathematical) is not established.
  • Limited to the last-token-only regime—scalars are not shared across tokens, so contextual encoding may not benefit.

Relevance to Vision-Language Models

This paper shows that spatial reasoning failures in VLMs stem largely from mis-calibrated internal activations rather than missing parametric knowledge, and that targeted reweighting of existing representations can recover most of the gap to full PEFT. It bridges activation steering / interpretability (identifying spatial-relevant heads) with practical parameter-efficient adaptation, offering a computationally cheap diagnostic and repair tool for VLM practitioners. For the broader VLM field, it reinforces the idea that frozen backbones contain latent spatial competence that can be unlocked with minimal intervention—a finding relevant to any multimodal reasoning task where the signal exists in the model but is dominated by competing modes.