Skip to content

M\(^3\)KG-RAG: Multi-hop Multimodal Knowledge Graph-enhanced Retrieval-Augmented Generation

🕒 Published (v1): 2025-12-23 07:54 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

M³KG-RAG proposes a graph-augmented RAG framework that builds a multi-hop multimodal knowledge graph (M³KG) covering audio, visual, and text modalities, then retrieves query-aligned subgraphs via modality-wise search and prunes them with GRASP (Grounded Retrieval And Selective Pruning). Evaluated on Audio-QA, Video-QA, and Audio-Visual QA, it consistently outperforms both text-KG and prior MMKG-based RAG baselines across multiple MLLMs including GPT-4o.

Problem

Existing multimodal RAG methods have two critical gaps: (1) current MMKGs are dominated by image-text pairs and lack audio-visual coverage with multi-hop connectivity, limiting temporal and causal reasoning; (2) retrieval via shared embedding spaces suffers from the modality gap—cross-modal distances are not calibrated, so queries retrieve off-topic or redundant knowledge that injects noise rather than improving answer quality.

Method

M³KG Construction (3-step multi-agent pipeline): - Context-Enriched Triplet Extraction: A rewriter agent augments raw multimodal captions with externally crawled knowledge (titles, descriptions), then an extractor LLM parses enriched captions into entity–relation triplets (h, r, t). - Knowledge Grounding: A normalizer canonicalizes entity surface forms; a searcher queries Wikipedia/Wiktionary for candidate descriptions; an LLM callback fills gaps for unsearchable concepts. - Context-Aware Description Refinement: A selector picks the most context-appropriate description per entity; a refiner adapts it to the original phrasing. An inspector agent implements a self-reflection loop, re-running uncertain outputs.

Each triplet in the resulting graph G is linked to at least one audio or visual item, guaranteeing full modality coverage for retrieval.

Modality-Wise Retrieval: Rather than cross-modal search, the system retrieves items in the same modality as the query (InternVL2 for video, CLAP for audio) via FAISS with a distance threshold Ï„, then lifts matched items to their connected multi-hop triplet subgraph.

GRASP: (1) Visual grounding with GroundingDINO scores entity presence across sampled frames; (2) Text-to-Audio Grounding (TAG) scores triplet presence in the audio stream; triplets below thresholds η are pruned. (3) A lightweight LLM (Qwen3-8B) applies a binary keep-or-drop policy over remaining triplets, discarding those not useful for answering the question.

Key Contributions

  • Multi-hop MMKG (M³KG) spanning audio, visual, and text with guaranteed per-triplet multimodal linkage, enabling modality-wise retrieval.
  • Lightweight 7-agent pipeline (rewriter, extractor, normalizer, searcher, selector, refiner, inspector) with self-reflection loop for automated, quality-controlled graph construction.
  • GRASP: two-stage pruning combining foundation-model grounding scores and LLM-based answer-relevance filtering to retain only query-relevant, answer-supportive triplets.
  • End-to-end M³KG-RAG framework integrating the above for graph-augmented generation in MLLMs.

Results

All scores are Model-as-Judge (M.J.) unless noted; baselines include None (no retrieval), Wikidata+naïve RAG, VTKG, M²ConceptBase, and VAT-KG.

  • AudioCaps-QA: VideoLLaMA2+M³KG-RAG: 53.23 vs. VAT-KG 44.60 vs. None 43.13. Qwen2.5-Omni+M³KG-RAG: 60.77 vs. VAT-KG 51.30 vs. None 49.00.
  • VCGPT (Video-QA): VideoLLaMA2: 39.92 vs. VAT-KG 39.42 vs. None 39.09. Qwen2.5-Omni: 44.35 vs. VAT-KG 43.50 vs. None 42.21.
  • VALOR (Audio-Visual QA): VideoLLaMA2: 29.25 vs. VAT-KG 28.30 vs. None 25.66. Qwen2.5-Omni: 44.67 vs. VAT-KG 35.44 vs. None 32.42.
  • GPT-4o on VALOR: 56.53 (M³KG-RAG) vs. 55.86 (VAT-KG) vs. 46.02 (None).
  • Win-rate vs. None (Qwen2.5-Omni, AudioCaps-QA): Overall 84.8% preference for M³KG-RAG.
  • Ablation on VALOR (Qwen2.5-Omni): Modality-wise retrieval only: 40.91; GRASP only: 36.96; both combined: 44.67; neither: 36.62.

Limitations

  • Benchmark-specific hyperparameters (Ï„, η) are tuned per dataset; generalization to unseen audio-visual distributions requires retuning.
  • Graph construction relies on external knowledge bases (Wikipedia, Wiktionary), creating coverage gaps for highly domain-specific or long-tail entities that fall back to a lightweight LLM callback.
  • Evaluation restricted to three benchmarks; Video-QA gains over baselines are modest (≤2 M.J. points), suggesting limited headroom in purely visual settings.
  • The self-reflection loop and multi-step agent pipeline add construction latency not quantified in the paper.
  • No analysis of graph size, retrieval latency, or computational cost at inference time.

Relevance to Agentic AI / LLM Agents

M³KG-RAG exemplifies multi-agent orchestration applied to a knowledge-intensive pipeline: seven specialized LLM agents (rewriter, extractor, normalizer, searcher, selector, refiner, inspector) collaborate with a self-reflection loop to automate quality-controlled knowledge graph construction, a pattern directly applicable to agentic memory and tool-use systems. The GRASP module demonstrates how grounding signals from foundation models can replace or augment LLM-as-judge pruning, a technique relevant to building more reliable agentic retrieval steps. For agents that must reason over heterogeneous sensory inputs (video, audio, text), M³KG-RAG's modality-aware retrieval and multi-hop graph traversal offer a principled alternative to flat embedding search. The work contributes to the broader line of research on structured external memory and tool-augmented reasoning for grounded, faithful agent responses.