Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

The paper analyzes how the projector in Multimodal LLMs (MLLMs) compresses visual information and aligns patch embeddings with LLM word embeddings, revealing that standard caption-loss training yields weak, coarse patch-level alignment. The authors propose Patch-Aligned Training (PAT), which adds a cosine-similarity loss between mask-selected vision patch embeddings and their corresponding LLM word embeddings during pretraining. This yields +16% on referring expression grounding, +4% on VQA, and +3% on instruction-following benchmarks over LLaVA-1.5.

Problem

Standard MLLM pretraining connects the vision encoder to the LLM through a projector trained only with image-caption loss. This coarse, global-level objective does not enforce patch-level correspondence between visual tokens and LLM word embeddings, leaving most visual tokens weakly or incoherently aligned with language. Prior analyses of fine-grained alignment lack quantitative metrics or improvement proposals that operate without adding inference-time tokens or auxiliary grounding modules.

Method

Analysis tools: 1. Von Neumann Entropy measures information compression across the projector. The normalized empirical covariance matrix of vision embeddings is used to compute spectral entropy; lower post-projection entropy indicates redundancy removal. 2. Patch-Level Localization (mIoU): For each object label in an image, average LLM subtoken embeddings to form a text embedding, then threshold cosine similarity with patch embeddings to predict object masks. IoU against ground-truth masks quantifies patch-level alignment Align(V, W). 3. Multi-Semantic Alignment via Matching Pursuit: Decomposes each vision patch embedding as a sparse linear combination of LLM word embeddings using orthogonal matching pursuit, revealing which semantic tokens each visual patch encodes.

Patch-Aligned Training (PAT): - An automated annotation pipeline (RAM → Grounding DINO → SAM) creates the Patch-Aligned Dataset (PAD) from the 558K LLaVA pretraining set, adding object tags, bounding boxes, and segmentation masks. - At pretraining, the patch-alignment loss L_patch = 1 − (1/P) Σ COS(v^(p), t^(p)) maximizes cosine similarity between mean patch embeddings within each mask and the corresponding LLM word embedding (averaged over subtokens). - Total loss: L = L_caption + β·L_patch, with β linearly increased from 0 to 5 during stage-1 pretraining. The patch loss bypasses LLM forward passes, making it computationally negligible relative to caption loss.

Key Contributions

• Formal characterization of projector compression using Von Neumann entropy; empirical demonstration that pretrained MLP projectors substantially compress visual features (entropy 4.84 → 2.04) while random projectors do not.
• Quantitative patch-level alignment metric (mIoU against mask-label annotations) showing LLaVA projectors achieve only 0.142 Align vs. 0.279 for PAT.
• Multi-semantic alignment hypothesis: each vision patch embedding is a sparse linear combination of LLM word embeddings, validated qualitatively via matching pursuit.
• Patch-Aligned Dataset (PAD): automatically annotated version of the 558K LLaVA pretraining set with dense mask-label pairs; pipeline and dataset publicly released.
• Patch-Aligned Training: lightweight pretraining objective that improves grounding without modifying inference architecture or adding tokens.

Results

• Referring Expression Comprehension (RefCOCO/+/g): +~16% average accuracy over LLaVA-1.5-7B (e.g., RefCOCO val: 56.22 → 65.97; RefCOCO+ test-B: 39.0 → 48.09).
• VQA benchmarks: GQA 61.93 → 63.0; SciQA 66.80 → 68.7; VizWiz 50.00 → 52.3; OKVQA 53.42 → 58.3 (all vs. LLaVA-1.5 baseline with same SFT).
• Instruction following (MMMU, MMVet, CMMMU, MMBench, MME): consistent small gains (e.g., MMMU 35.30 → 36.56; MMVet 30.70 → 31.61; MME Perception 1510.75 → 1531.33).
• Caption quality (stage-1 only, frozen LLM): METEOR 0.1220 → 0.1256; ROUGE-L 0.1661 → 0.1759; SPICE 0.1571 → 0.1710.
• Patch alignment: Align(V,W) 0.142 → 0.279; Von Neumann entropy reduction ΔH 2.80 → 3.84.
• Gains generalize to LLaMA-3.1-8B backbone and C-Abstractor projector (Table 9).

Limitations

• The multi-semantic alignment hypothesis lacks ground-truth multi-label per-patch supervision; validation is qualitative only.
• Averaging word embeddings across subtokens to form t^(p) is a coarse representation; polysemous and compositional semantics may be poorly captured.
• Over-compression (large β) causes performance degradation by discarding semantically useful information, and the optimal β requires tuning.
• The annotation pipeline depends on RAM, Grounding DINO, and SAM accuracy; noisy auto-labels propagate into the patch loss.
• Evaluated only on LLaVA-1.5 scale (7B); scalability to larger models or higher-resolution inputs is not demonstrated.

Relevance to Vision-Language Models

This paper directly addresses a core architectural question in VLMs: what role does the vision-language projector play beyond a simple modality bridge? By introducing formal compression (Von Neumann entropy) and patch-alignment (mIoU, matching pursuit) diagnostics, it provides analytical tools applicable to any connector-based MLLM architecture. The finding that caption-loss training leaves most visual tokens misaligned with LLM vocabulary helps explain persistent hallucination and weak grounding in models like LLaVA, and the proposed patch-alignment loss offers a drop-in pretraining improvement requiring no inference-time changes. For VLM researchers, this work motivates rethinking the pretraining objective at the token level rather than the image level.