Skip to content

ChainMPQ: Interleaved Text-Image Reasoning Chains for Mitigating Relation Hallucinations

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Interleaved text-image reasoning chains mitigate relation hallucination in VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ChainMPQ is a training-free framework that mitigates relation hallucinations in LVLMs—where models correctly detect objects but misidentify their relationships—by decomposing each relational query into five multi-perspective sub-questions and building an interleaved text-image reasoning chain that passes accumulated textual answers and visual attention masks across steps. Relation hallucinations account for ~40% of all LVLM hallucinations yet have received far less attention than object or attribute hallucinations. ChainMPQ consistently reduces relation hallucinations across four architectures and two benchmarks without any fine-tuning.

Problem

LVLMs frequently hallucinate relations between correctly-identified objects (e.g., predicting "standing on" when the true relation is "riding"), a failure mode comprising ~40% of all LVLM hallucinations. Prior mitigation work targets object/attribute hallucinations; the few relation-focused methods (constrained prompting, output-layer calibration, triplet conversion) treat relational reasoning as a single inference step, causing over-reliance on language priors and neglect of visual evidence. No method exploits accumulated multimodal context across multiple reasoning steps.

Method

ChainMPQ operates in three sequential modules applied at inference time:

1. Text-guided Attention Enhancement. Subject and object keywords are extracted from the question via spaCy and encoded as \(X \in \mathbb{R}^{N \times d_t}\). Cross-attention between visual patch features \(V \in \mathbb{R}^{M \times d_v}\) (query) and keyword embeddings (key/value) produces enhanced visual tokens: $\(V' = \text{softmax}\!\left(\frac{VX^\top}{\sqrt{d_t}}\right)X\)$ that emphasize entity-relevant image regions.

2. Multi-Perspective Aware Text Prompt. The original question is decomposed into three semantic components—subject \([S]\), object \([O]\), relation \([R]\)—and five sub-questions are generated: \(Q_1\)/\(Q_2\) localize \([S]\) and \([O]\); \(Q_3\) masks \([O]\) to ask what \([S]\) interacts with; \(Q_4\) masks \([S]\) to ask what \([O]\) is affected by; \(Q_5\) masks \([R]\) to ask for the general relationship.

3. Interleaved Text-Image Reasoning Chain. \(Q_1\)–\(Q_5\) are input sequentially. From \(Q_3\) onward, attention weights over keyword tokens from the last \(n\) decoder layers identify the top-\(k\) most-attended visual tokens, with \(k\) selected adaptively via normalized entropy: $\(k = k_{\max} \cdot \hat{H}(\text{Attn}_i), \quad k_{\max} = 20\)$ These tokens form a bias mask \(M_i\); subsequent attention is modified as: $\(\text{Attn}_{i+1} = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}} + \alpha_i \cdot M_i\right)V\)$ where \(\alpha_i = \lambda \cdot \text{Conf}_{\text{prev}_i}\) scales the bias by the confidence of the preceding answer. Multi-round visual biases are averaged by confidence weight. Textual answers accumulate as context \(C_i\) for each subsequent question. The original question is then answered using the full accumulated text context and visual memory.

Key Contributions

  • Subject–object–relation decomposition generating five systematic multi-perspective sub-questions that isolate each relational component before synthesis.
  • Entropy-adaptive top-\(k\) selection of visual tokens from decoder attention maps, transferred as attention biases across reasoning steps to maintain visual grounding.
  • Confidence-weighted accumulation of multi-round visual biases, progressively refining attention focus.
  • Demonstration of consistent, training-free relation hallucination reduction across four architectures (LLaVA-1.5, InstructBLIP, Qwen2.5-VL, InternVL3.5) and two benchmarks.
  • Two lightweight variants (Light1, Light2) trading modest accuracy for substantially lower latency.

Results

  • MMRel, LLaVA-1.5: ChainMPQ 65.20% acc vs. best baseline (Calibrate) 63.50%; F1 71.21 vs. 70.54.
  • MMRel, InstructBLIP: 65.14% acc vs. 64.52% (Prompting); F1 74.12 vs. 71.23.
  • MMRel, Qwen2.5-VL: 73.52% acc vs. 71.36% (Calibrate); F1 77.45 vs. 76.28.
  • MMRel, InternVL3.5: 75.21% acc vs. 74.45% (Calibrate); F1 78.65 vs. 78.64.
  • R-Bench, LLaVA-1.5: 76.04% acc, 72.03% prec (+4.17% prec over best baseline Prompting at 67.86%); F1 81.54 vs. 79.60.
  • R-Bench, Qwen2.5-VL: 83.92% acc, 80.23% prec, 84.63% F1—best across all models.
  • Latency (LLaVA-1.5, MMRel): Full ChainMPQ: 3.3 s/sample, \(\Delta\)Acc/\(\Delta\)Time = 2.58; Light1 (Q1,Q2,Q5 parallel): 1.5 s/sample, ratio = 7.93; Vanilla: 0.9 s/sample.
  • Ablation (LLaVA-1.5, MMRel): Removing multi-perspective questions: −3.68% acc; removing interleaved chain: −3.08%; removing text-guided attention: −1.14%.

Limitations

  • Inference latency increases 3.7× over vanilla (3.3 s vs. 0.9 s per sample on LLaVA-1.5); lightweight variants partially recover efficiency but sacrifice accuracy.
  • Text-guided attention enhancement contributes only +1.14% accuracy, suggesting keyword-based cross-attention is a weak link in the pipeline.
  • Evaluation restricted to binary Yes/No VQA relation benchmarks (MMRel, R-Bench); generalization to open-ended relational generation tasks is not assessed.
  • Only 7B-scale open-source models evaluated; behavior on larger or closed-source LVLMs is unknown.
  • Keyword extraction relies on spaCy dependency parsing, which may fail on ambiguous or complex questions.
  • No evaluation on object/attribute hallucination benchmarks to confirm the method does not regress general multimodal performance.

Relevance to Vision-Language Models

ChainMPQ directly addresses an underexplored but quantitatively dominant failure mode in LVLMs—relation hallucinations—by introducing structured, multi-step inference that grounds relational reasoning in both textual and visual evidence. It advances the line of multimodal chain-of-thought research (ICoT, CoT) by making visual attention state an explicit transferable memory across reasoning steps, complementing text-only prompting strategies. The training-free, architecture-agnostic design makes it immediately applicable to current LVLM deployments and offers a modular template for addressing other structured reasoning failures in multimodal models. For researchers tracking VLMs, this work highlights inference-time decomposition as a practical alternative to expensive fine-tuning for targeted hallucination mitigation.