Skip to content

From Heads to Neurons: Causal Attribution and Steering in Multi-Task Vision-Language Models

🕒 Published (v1): 2026-04-20 08:21 UTC · Source: Arxiv · Venue: ACL 2026 · link

Why this paper was selected

Neuron-level causal attribution reveals cross-task VLM mechanisms; ACL 2026

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HONES is a gradient-free framework that attributes task-critical FFN neurons in multi-task VLMs by first localizing task-relevant attention heads via causal interventions, then scoring neurons by their write-in contribution conditioned on those routing heads. It also introduces lightweight inference-time steering via sparse neuron-wise scaling factors, achieving consistent performance gains across four heterogeneous vision-language tasks on two VLMs.

Problem

Existing neuron-level interpretability methods for VLMs have two key gaps: (1) they operate on single tasks, making cross-task neuron comparability impossible, and (2) they rank neurons in isolation without accounting for how task-relevant attention heads route information, which inflates importance scores for polysemantic neurons and yields noisy task-critical sets in multi-task settings.

Method

HONES is a two-stage, gradient-free framework operating on frozen VLM backbones.

Stage 1 — Critical Head Localization. For each task \(t\), HONES adopts V-SEAM, a mean-replacement causal intervention: head \(h\)'s output \(o^{(h)}(x)\) is replaced by the mean of the other \(H-1\) heads. A head importance score \(S_t(h) = \mathbb{E}_{(x,y)\sim D_t^{\text{disc}}} \Delta P_t(x,y; I_h)\) measures expected task-utility degradation. The top-\(K_h\) heads form \(H_t^*\).

Stage 2 — Head-Guided Neuron Attribution. For each FFN neuron \((l, i)\), the sample-level write-in contribution is computed via Direct Vocabulary Projection (DVP): $\(c_{l,i}(x,y;\theta) = \langle \Delta r_i^{(l)}, u_y \rangle, \quad \Delta r_i^{(l)} = z_i^{(l)} W_{\text{down}}^{(l)}[i,:]\)$ where \(u_y\) is the target unembedding vector (mean-normalized for fixed-set targets; IDF-weighted for open-ended captioning). For each critical head \(h \in H_t^*\), HONES re-runs the forward pass under head intervention \(I_h\) and measures the contribution drop \([\Delta c_{l,i}^{(h)}(x,y)]_+\). A head-importance-weighted aggregate across heads and data gives the final neuron importance \(I_{l,i}\); top-\(K\) neurons form \(N_t^*\).

Stage 3 — Steering. Per-neuron learnable scaling factors \(\lambda_{l,i}\) are trained on \(D_t^{\text{dev}}\) (backbone frozen) by minimizing task loss plus a KL-divergence regularizer \(\beta \cdot \text{KL}(p_\theta(\cdot|x) \| p_{\theta_{\lambda_t}}(\cdot|x))\) to keep the scaled model close to the original.

Key Contributions

  • HONES: a unified, gradient-free, cross-task neuron attribution framework that conditions FFN neuron scores on task-relevant attention head routing rather than ranking neurons in isolation.
  • Empirical finding that task-critical neurons concentrate in middle (layers 10–20) and deep (>25) layers, with retrieval peaking earlier and the other tasks peaking deeper.
  • Cross-task neuron overlap analysis showing that shared neurons (especially those overlapping with VQA) are more causally important than task-specific ones.
  • Lightweight task-specific steering via sparse scaling factors on \(N_t^*\) that improves performance across all four tasks without modifying backbone weights.

Results

Neuron localization (causal importance = relative performance drop after masking top-1% neurons): - HONES achieves 18.38% average drop on LLaVA-1.5-7B and 21.91% on Qwen2.5-VL-7B across four tasks. - Best activation-based baseline (MA): 8.89% / 12.02%; best causal baseline on VQA (QRNCA): 20.80% / 24.50% vs. HONES 27.30% / 36.50%. - Ablation HONES-NoHead (no head conditioning): 5.33% / 8.21% — head conditioning is responsible for most of HONES's gain. - Entropy-based APE can yield negative drops (performance improvement), indicating it selects task-irrelevant neurons.

Steering (results truncated in text; table partially shown): HONES steering outperforms fixed amplification, grid-searched amplification, random-neuron scaling, and HONES-without-KL across both VLMs on all four tasks.

Generalization: Trends hold on LLaVA-1.5-13B (full results in appendix).

Limitations

  • The paper restricts analysis to two VLM architectures (LLaVA-1.5-7B and Qwen2.5-VL-7B); generalization to other architectural families (e.g., encoder-decoder models, models with cross-attention) is unconfirmed.
  • Top-\(K_h\) and top-\(K\) hyperparameters (30 heads for LLaVA, 25 for Qwen; 1% of neurons) require empirical tuning per model and task.
  • The neuron definition is tied to the intermediate activation \(z^{(l)}\) of the last token in autoregressive decoding; applicability to encoder-decoder VLMs or bidirectional models is not discussed.
  • Steering requires a separate labeled validation split \(D_t^{\text{dev}}\) per task; the approach is not zero-shot.
  • The steering performance table is partially truncated in the provided text, limiting full quantitative assessment of gains.

Relevance to Vision-Language Models

HONES directly advances mechanistic interpretability of multi-task VLMs by revealing that task-critical computation is organized through sparse attention-routing pathways that feed into specific FFN neuron subsets — a finding with immediate implications for model debugging, capability localization, and targeted editing. The cross-task neuron overlap analysis (VQA-shared neurons being most salient) is particularly relevant for understanding how shared VLM parameters generalize across heterogeneous vision-language tasks. The gradient-free design and vectorized scoring make HONES tractable for large VLMs where fine-grained patching is prohibitive, offering a practical alternative to SAE-based sparse dictionary approaches. For researchers tracking VLMs, HONES provides both an attribution tool and a lightweight steering mechanism that achieves task-specific gains without fine-tuning the backbone.