PostAlign: Multimodal Grounding as a Corrective Lens for MLLMs

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Philip Torr/Oxford; multimodal grounding corrects MLLM fine-grained visual failures

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MLLMs hallucinate partly because linguistic priors dominate visual evidence at inference time — an 89.2% overlap in hallucinated tokens persists even when the image is removed. MMGrounded-PostAlign adds a plug-in post-alignment framework on top of frozen MLLMs that forces outputs to be anchored by explicit pixel-level grounding (mask + bounding box) and adaptive textual rationales. Applied to six backbones (LLaVA-1.5, Qwen2-VL, Qwen2.5-VL, InternVL3/3.5), it improves hallucination and VQA benchmarks without degrading general reasoning.

Problem

MLLMs over-rely on language co-occurrence statistics rather than actual visual content, producing hallucinations (fabricated objects, wrong attributes, spurious spatial relations). Prior mitigations are either post-hoc decoding tricks with high inference overhead, or training-data augmentations that address symptoms rather than the grounding deficit. No prior work repurposes grounding outputs as a lightweight corrective signal applied after the main MLLM without redesigning the grounding architecture.

Method

MMGrounded-PostAlign is a LoRA-based fine-tuning framework built on top of existing MLLM backbones. It introduces a structured output format \(A = \{V, T, F\}\) — a visual grounding token, a textual grounding token, and the final answer — generated jointly by the MLLM.

Visual grounding module. The last-layer embedding of a special <LOC> token is projected via MLP and used as a prompt for a frozen ViT-H SAM encoder + multi-task decoder, which produces both a segmentation mask (\(L_\text{seg}\) = BCE + DICE) and a bounding box (\(L_\text{det}\) = Smooth-L1 + GIoU). These grounding outputs act as implicit constraints on the final answer generation.

Negative rejection mechanism. When the queried referent is absent from the image, the MLLM is trained to emit <REJ> instead of <LOC>. The decoder assigns it an empty mask/box, and a binary cross-entropy rejection loss \(L_\text{rej}\) penalizes incorrect grounding of non-existent objects.

Selective reasoning mechanism (textual grounding). A self-reflection prompt at inference time routes each query into <SIMPLE> (answer directly) or <COMPLEX> (generate a rationale first). A binary cross-entropy loss \(L_\text{reason}\) supervises this routing. The combined training objective is: $\(\mathcal{L} = \lambda_1 L_\text{rej} + \lambda_2 L_\text{reason} + L_\text{ground} + L_\text{text}\)$

Only LoRA weights, the multi-task decoder, LLM token embeddings, LLM head, and projection layer are updated; the visual grounding encoder is frozen.

Key Contributions

• MMGrounded-PostAlign framework: a post-alignment module applicable to any MLLM backbone without changing the grounding architecture.
• Negative rejection mechanism: explicit <REJ> token + loss that teaches the model to refuse grounding of hallucinated objects.
• Selective reasoning mechanism: <SIMPLE>/<COMPLEX> routing that invokes intermediate rationale generation only when query complexity warrants it.
• Empirical analysis of linguistic priors: layer-wise token probability analysis and image-removal ablation showing 89.2% persistence of hallucinated tokens confirms that priors dominate visual signals.
• Consistent gains across six MLLM backbones (7B–14B) without degrading general reasoning capability.

Results

• POPE (hallucination) — LLaVA-1.5-7B baseline: Random/Popular/Adversarial = 83.3/80.1/78.2 → PostAlign: 86.6/84.2/82.3; LLaVA-1.5-13B: 85.4/82.2/79.2 → 88.9/87.3/85.6; Qwen2-VL-7B: 88.9/86.8/84.6 → 90.3/89.2/87.1; InternVL3-14B: 89.1/87.2/84.3 → 91.2/89.0/86.6.
• HaloQuest (ablation) — Adding <REJ> to full grounding lifts False Premise Human Eval from 9.9 → 33.2 and Insufficient Context from 9.9 → 31.4 (LLaVA-1.5-7B).
• MME — InternVL3.5: 1792.2 → 1906.9; gains across all backbones.
• MMBench EN/CN — Consistent +1–2 pt gains across all backbones.
• Explicit grounding vs. bounding-box-as-text (BTL) — PostAlign outperforms both BTL-Generation and BTL-Caption on POPE, MME, and MMBench for both 7B and 13B LLaVA models.
• Results on RefCOCO grounding and ReasonSeg are mentioned but numerical details for those benchmarks are cut off in the provided text.

Limitations

• Requires fine-tuning with LoRA plus a multi-task grounding decoder; not a pure inference-time method, so it needs annotated grounding data.
• Built on SAM ViT-H as the visual grounding backbone; performance may depend on SAM's own localization quality.
• Query complexity routing (<SIMPLE>/<COMPLEX>) relies on self-reflection prompting and binary loss, which may misclassify edge-case queries.
• Full numerical results on RefCOCO and ReasonSeg grounding benchmarks are not shown in the excerpt, so the grounding accuracy trade-offs are not fully assessable.
• Training data construction (annotation of <SIMPLE>/<COMPLEX>/<REJ>) adds pipeline complexity not fully detailed in the main text.

Relevance to Vision-Language Models

PostAlign directly addresses one of the core open problems in VLM research — the dominance of language priors over visual grounding — with a modular, backbone-agnostic solution that can be applied as a post-training "lens" on top of existing MLLMs. The negative rejection mechanism is particularly relevant for safety-critical VLM deployment scenarios where fabricated objects are unacceptable. The selective reasoning mechanism connects to the broader chain-of-thought literature for MLLMs, showing that adaptive rationale generation (rather than always-on or always-off) yields better accuracy and efficiency. For those tracking the hallucination mitigation subfield, this paper provides both a diagnostic (the 89.2% linguistic-prior dominance finding) and a practical recipe applicable to frontier-scale models (InternVL3.5-14B).