Learning to Steer: Input-dependent Steering for Multimodal LLMs

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

L2S (Learn-to-Steer) replaces static mean-steering vectors in multimodal LLMs with input-dependent steering vectors predicted by a lightweight 2-layer MLP auxiliary network. The auxiliary network is trained to approximate contrastive prompt-derived (P2S) steering vectors from the input's latent representation, enabling behavior-specific steering without requiring the contrastive prompts at inference time. L2S outperforms all static steering baselines on both safety enforcement and hallucination mitigation while adding negligible computational overhead.

Problem

Existing steering methods for LLMs/MLLMs (e.g., mean steering / CAA) apply a single fixed steering vector regardless of input, which is inadequate when the desired behavior is input-dependent — e.g., "safe" responses may mean refusing an illegal query or recommending an expert for medical advice, requiring distinct steering directions. Furthermore, steering for MLLMs has been under-explored compared to text-only LLMs.

Method

P2S (Prompt-to-Steer): For each input X=(I,T), construct input-specific contrastive prompts T⁺X (desired behavior) and T⁻_X (undesired behavior), appended to the base prompt and run in teacher-forcing mode through the LLM. The steering vector z{X,L} is the difference between the last-token hidden representations at layer L for the positive and negative augmented inputs. At inference, apply z_{X,L} as a linear shift to residual stream representations at layer L with magnitude α. P2S is an oracle method requiring known contrastive prompts at test time.

L2S (Learn-to-Steer): Train a lightweight 2-layer MLP g_Θ: ℝ^D → ℝ^D to predict P2S steering vectors from the input context representation h_{X,L'} (last-token hidden state at an intermediate layer L' just before generation). Training minimizes MSE: Θ = argmin_Θ E_X[||z_{X,L} − g_Θ(h_{X,L'})||²₂]. At inference, the predicted vector g_{Θ}(h_{X,L'}) is injected as a residual shift at layer L for all generated tokens. The MLP has hidden size 100 and is trained for 100 epochs on a single RTX 5000 GPU. No access to model weights is needed after representation extraction; g_Θ trains purely in representation space.

Key Contributions

• Identifies the input-dependence limitation of static steering vectors and demonstrates it empirically on safety and hallucination tasks.
• Proposes P2S, a training-free input-specific steering method using contrastive prompt pairs; serves as a strong oracle upper bound.
• Proposes L2S, which learns to predict P2S steering vectors via a tiny MLP auxiliary network, making input-dependent steering practical at inference time.
• Demonstrates that L2S outperforms mean steering, prompting-based baselines, and noise-injection baselines on MMSafetyBench and POPE, applied to LLaVA-v1.5-7B and Qwen2-VL-7B.

Results

Safety (MMSafetyBench, LLaVA-v1.5): - L2S Unsafe-score (p≥0.5/0.7/0.9): 0.082 / 0.057 / 0.034 vs. Mean-S(BA): 0.089 / 0.066 / 0.041 vs. No-steering: 0.276 / 0.234 / 0.204 - L2S ED-score (expert deferral): 0.395 vs. Mean-S: 0.329 vs. No-steering: 0.250 - Response quality: 6.56 (L2S) vs. 6.92 (No-steering) — modest quality drop

Safety (Qwen2-VL): - L2S Unsafe-score (p≥0.5): 0.058 vs. Mean-S(BA): 0.046 (Mean-S(BA) slightly better on Unsafe); L2S ED-score: 0.592 vs. Mean-S: 0.408 - Response quality: 7.78 (L2S) vs. 7.23 (No-steering)

Hallucination (POPE, LLaVA-v1.5, Random subset): - L2S: Acc 86.46%, F1 92.74% vs. Mean-S: 84.29% / 91.47% vs. No-steering: 82.73% / 90.55% - Adversarial subset: L2S 77.76% / 87.48% vs. No-steering 76.82% / 86.89% (smaller but consistent gain)

CHAIR (LLaVA-v1.5, 500 COCO images): - L2S: CHAIRs 16.10 / CHAIRi 51.80 vs. No-steering: 17.31 / 52.80 - Gemini Win Rate: L2S 64.20% vs. No-steering 35.80%

Limitations

• Contrastive prompt pairs must be designed manually per behavior category; no guarantee the chosen pairs are optimal from the vast space of possible completions.
• Steering is applied at a single residual stream layer via a linear shift; multi-layer or non-linear interventions could improve effectiveness.
• L2S requires a labeled training set with known behavior categories to construct P2S vectors; unseen behavior types at training time may not generalize.
• Dual-use risk: the same mechanism could steer models toward harmful behaviors if misused.
• POPE gains on adversarial subset are modest relative to the P2S oracle gap, suggesting harder hallucination scenarios are only partially addressed.

Relevance to Vision-Language Models

L2S directly addresses a core reliability challenge for MLLMs — that safety and grounding behaviors are not monolithic and vary with input context — by introducing a lightweight, post-hoc steering mechanism that avoids expensive fine-tuning. It builds on the linear representation hypothesis applied specifically to the multimodal decoder, demonstrating that a tiny learned module can approximate complex input-conditional behavioral shifts at essentially no inference cost. For researchers tracking VLMs, this is relevant as a complementary alignment technique to RLHF/DPO, particularly useful when model weights are fixed or fine-tuning is cost-prohibitive. The contrastive prompting framework also connects to activation engineering and mechanistic interpretability threads in the VLM literature.