Skip to content

MODE-RAG: Manifold Outlier Diagnosis and Energy-based Retrieval-Augmented Generation Evaluation

🕒 Published (v1): 2026-06-16 03:06 UTC · Source: Arxiv · Venue: ACL 2026 · link

Why this paper was selected

ACL 2026; multimodal RAG evaluation exposing cross-modal hallucination and sycophancy failures

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MODE-RAG is a multi-agent framework that resolves the "intervention paradox" in Multimodal RAG (M-RAG) systems—where static correction rules either over-correct accurate outputs or under-correct hallucinated ones—by using Variational Free Energy (VFE) to gate interventions dynamically. A five-agent pipeline handles high-risk queries with MCTS-guided causal reasoning and logit perturbation to suppress sycophancy. The authors also introduce ModeVent, a 1,000-sample adversarial benchmark derived from MultiVent by selecting the 500 highest- and 500 lowest-VFE samples.

Problem

Multimodal RAG systems built on Large Vision-Language Models suffer from cross-modal hallucinations, causal fabrication, and sycophancy when retrieved context conflicts with visual evidence. Existing mitigations face an intervention paradox: rule-based constraints over-correct accurate outputs, while unguided multi-step reasoning causes formatting instability and cascading logical failures. No prior work dynamically decides when intervention is warranted based on model-internal uncertainty signals.

Method

FE-Router (Thermodynamic Gate): For each query the router computes the Variational Free Energy (VFE) over the model's predictive distribution:

\[F(q, x; \tau) = \sum_{j=1}^{|V|} q(j)\left[-f_j(x) + \tau \log q(j)\right]\]

If \(\bar{F} > \gamma\), the query is routed to the five-agent pipeline; otherwise it bypasses to avoid over-correction. Internal attention states are monitored via ATLAS (Attention-based Transparent Latent Assessment System).

Five-Agent Pipeline (activated only for high-risk queries): 1. Per-Agent — extracts symbolic atomic-fact triplets \(\langle s, p, o \rangle\) from the visual stream as a "Grounded Truth Anchor." 2. Cor-Agent — wraps all inter-agent data in a strict JSON-Schema (Facade Pattern) to prevent semantic noise leakage. 3. Ret-Agent — scores each retrieved document \(d_i\) by manifold fidelity in a whitened latent space: \(\text{Score}(d_i) = \text{Sim}_i \cdot S_i \cdot I_i\), discarding documents that fall in high-energy outlier regions. 4. Rea-Agent — runs MCTS with UCT selection to build a Causal DAG over the visual evidence, explicitly preventing temporal inversion and imposed causality. 5. Gen-Agent — synthesizes output and applies logit perturbation at decoding time to penalize hallucination-aligned tokens and boost DAG-aligned tokens.

Overseer (PORAG): Performs a triple-consistency check across the Per-Agent triplets, Rea-Agent DAG, and Gen-Agent text. Failures trigger a recursive fallback that increases MCTS simulation budget; unresolved conflicts produce an explicit Epistemic Refusal.

ModeVent Benchmark: SigLIP/CLIP features are extracted from all MultiVent samples, whitened for isotropy, and ranked by mean VFE. The top-500 (manifold outliers, high VFE) and bottom-500 (stable inliers, low VFE) form the benchmark. Adversarial contexts covering 7 RAG error categories are synthesized via DeepSeek-V3.2 guided by Qwen3-Omni-30B + Florence-2 ground-truth fusions.

Key Contributions

  • FE-Router: VFE + ATLAS-driven adaptive gate that resolves the intervention paradox by bypassing accurate queries and intercepting high-risk ones.
  • Decoupled five-agent pipeline with MCTS-guided causal DAG construction and logit-level sycophancy suppression.
  • PORAG Overseer with recursive fallback and Epistemic Refusal for unresolvable conflicts.
  • ModeVent benchmark: 1,000-sample adversarial M-RAG evaluation set (500 manifold outliers + 500 stable inliers) across 7 RAG error categories, evaluated with LLM-as-a-Judge on Fidelity (F) and Resilience (R).

Results

All experiments use Qwen-2.5-VL-7B as the base VLM kernel; evaluation via DeepSeek-V3.2 LLM-as-a-Judge on 0–5 Fidelity and 0–5 Resilience scales (Total = F + R, max 10).

  • Global average Total Score: +1.04 improvement (4.40 → 5.45).
  • Global Fidelity gain: \(\Delta F = +0.89\); Resilience gain: \(\Delta R = +0.16\) — improves factual accuracy without meaningfully sacrificing completeness.
  • Outlier scenarios (hardest): \(\Delta\text{Total} = +1.48\) overall; largest gains on Majority Text Bias (\(\Delta\text{Total} = +2.31\)) and Out-of-Domain Irrelevance (\(\Delta\text{Total} = +1.68\)).
  • Inlier scenarios: \(\Delta\text{Total} = +0.60\); Information Sparsity shows \(\Delta\text{Total} = +0.93\).
  • Suppression of catastrophic failures (score 0–2) in categories like Majority Text Bias and Metadata Redundancy.
  • Comparison against Self-RAG, SelfCheckGPT, and Woodpecker also reported (full table in Appendix B; not included in truncated text).

Limitations

  • Hardware cost: MCTS expansion and multi-agent inference require 4× RTX 4090 GPUs, making the full pipeline expensive relative to a single-pass VLM.
  • ModeVent is derived from a single source dataset (MultiVent), limiting domain diversity of the benchmark.
  • Adversarial contexts are synthetically generated by DeepSeek-V3.2; real-world retrieved noise may differ in character.
  • The VFE threshold \(\gamma\) requires calibration on held-out data; robustness of threshold selection across domains is not analyzed.
  • Resilience can drop in inlier categories where aggressive pruning discards relevant text (e.g., Causal Imposition inliers: \(\Delta R = -0.39\)).

Relevance to Vision-Language Models

MODE-RAG directly targets a critical failure mode of deployed VLMs in retrieval-augmented settings: the inability to adjudicate conflicts between retrieved text and visual evidence. The use of VFE over model logits as a hallucination risk signal is a mechanistic, model-internal approach applicable to any instruction-tuned VLM kernel, making it architecturally general. The MCTS-based causal DAG construction and logit perturbation at decoding are test-time interventions that do not require fine-tuning, relevant for practitioners adapting off-the-shelf VLMs. ModeVent also fills a benchmark gap—existing M-RAG evaluations lack adversarially constructed cross-modal conflict cases stratified by model uncertainty.