Learning to Instruct for Visual Instruction Tuning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

L2T (Learning to InstrucT) augments standard visual instruction tuning (VIT) by extending the training loss to also predict the instruction sequence (minus boilerplate template tokens), forcing the model to proactively attend to image content. This acts as a zero-cost regularizer: no extra data, <1% compute overhead, yet yields up to 8.5% overall improvement across 16 benchmarks and up to 17.6% on captioning.

Problem

Standard VIT trains MLLMs solely to predict response tokens given image + instruction, creating an incentive to exploit language priors while ignoring visual input — a shortcut learning failure demonstrated by showing a pure LLM (no image) achieves correct answers on many VQA samples. This leads to overfitting, degraded visual grounding, and hallucinations.

Method

L2T modifies the VIT fine-tuning loss to jointly maximize likelihood of both the instruction and the response conditioned on the image:

\[\mathcal{L} = -\log p_\theta(X_I, X_A | X_V) = \underbrace{-\sum_i \log p_\theta(X_{I,i}|X_V, X_{I,<i})}_{\text{Learn to Instruct}} \underbrace{-\sum_i \log p_\theta(X_{A,i}|X_V, X_I, X_{A,<i})}_{\text{Learn to Respond}}\]

Template removal filters two classes of tokens from the instruction learning signal: (1) system templates (USER/ASSISTANT role markers, assistant persona tokens) and (2) task templates — high-frequency boilerplate sentences identified by corpus-wide frequency counting (e.g., "Answer the question using a single word or phrase"). Only semantically image-relevant instruction content is learned. The method is applied exclusively during fine-tuning, not pre-training, and requires no new training data.

To verify the mechanism, the authors define visual contribution (VC) = log p(XA|XV, XI) − log p(XA|XV=∅, XI), showing L2T achieves a 9% higher VC, confirmed by attention weight visualizations showing stronger activation on visual tokens.

Key Contributions

• A minimal modification to the VIT objective — adding instruction-prediction loss — that regularizes against shortcut learning without any data augmentation or architectural change.
• Automated frequency-based template removal to ensure instruction learning targets image-relevant content only.
• Comprehensive evaluation across 16 benchmarks on TinyLLaVA (0.5B, 3B), LLaVA-1.5 (7B, 13B), LLaVA-NeXT (7B), and Prism-7B showing consistent gains.
• Demonstrated reduction in hallucinations across four hallucination benchmarks (POPE, CHAIR, MMHAL-Bench, HallusionBench).
• Pilot of self-improving instruction tuning: L2T model generates its own 100k instruction–response pairs for bootstrapped re-training, yielding further gains.

Results

• Overall (16 benchmarks): +6.1% to +8.5% relative improvement over VIT depending on model; best on LLaVA-NeXT Vicuna-7B (+8.5%).
• OCR/Chart/Doc understanding: avg +4.6% to +8.8% relative; DocVQA: 21.43→25.59 on LLaVA-1.5 13B; ChartQA: 54.88→66.80 on LLaVA-NeXT.
• Image captioning: avg +11.5% to +17.6% relative; RefCOCO: 17.50→56.16 on TinyLLaVA Qwen2-0.5B; largest gains here across all categories.
• Hallucination (LLaVA-1.5 7B): MMHAL-Bench GPT score 1.73→2.36 (+36%), hallucination rate 0.68→0.53 (−22%); CHAIR-s 48.60→46.20 (−2.4 pp), CHAIR-i 13.40→11.80 (−1.6 pp).
• Prism-7B generalization: RefCOCO 56.70→66.00, VSR 53.20→61.70, TextVQA 52.80→55.60.
• Text-only benchmarks: MT-Bench 5.35→5.49, MMLU 49.57→49.18 (no degradation).
• General VQA (VQAv2, GQA, ScienceQA, VizWiz): modest improvements of +0.2% to +1.5% on average.
• Compute: L2T averages 0.331 steps/sec vs. VIT's 0.334 steps/sec (<1% overhead).

Limitations

• Gains are instruction-quality dependent: grounding data (instructions are fixed templates + bounding boxes) yields ~0% improvement, exposing a ceiling tied to instruction informativeness.
• Template removal relies on heuristic frequency thresholding; unusual instruction datasets may require custom thresholds.
• LLaVA-NeXT results are not directly comparable to official numbers because the training script is partially closed-source.
• Self-improving pipeline is a pilot; no ablation on the quality or diversity of self-generated instructions.
• Not benchmarked on frontier-scale models (>13B) or recent proprietary VLMs, limiting extrapolation to state-of-the-art systems.

Relevance to Vision-Language Models

L2T directly addresses a structural flaw in the dominant VIT paradigm — that response-only supervision allows models to bypass visual input — which is a root cause of both hallucination and poor visual grounding, two major open problems in VLM research. Because the method is a loss-level change orthogonal to architecture, data curation, and model scale, it can be composed with any VIT-based MLLM (demonstrated on LLaVA-1.5, LLaVA-NeXT, Prism). The captioning results (up to +17.6%) are particularly notable, as captioning quality is a proxy for holistic visual understanding and frequently used in VLM pre-training evaluation. The self-improving pilot opens a direction toward data-flywheel approaches for VLMs where the model's own instruction generation capability feeds back into training.