Skip to content

Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Wiki-PRF is a three-stage (Processing–Retrieval–Filtering) retrieval-augmented generation framework for knowledge-based VQA that trains a VLM via reinforcement learning to dynamically invoke visual tools for fine-grained query construction and to filter retrieved Wikipedia content down to task-relevant knowledge. It achieves state-of-the-art performance on E-VQA (36.0) and InfoSeek (42.8), and is the first RL-based method for multimodal RAG.

Problem

Existing RAG-based KB-VQA methods suffer from two compounding failures: (1) coarse retrieval—whole-image features cause salient but irrelevant objects (e.g., a church behind a statue) to dominate retrieval queries, missing the actual subject; and (2) imprecise filtering—passage-level reranking retains large amounts of noise because it cannot reason about which sub-sentences are answer-relevant given the question.

Method

Wiki-PRF operates in three sequential stages, all coordinated by a RL-trained VLM called VLM-PRF (Qwen2.5-VL 3B or 7B fine-tuned with LoRA via GRPO without KL divergence):

  1. Processing stage: VLM-PRF reasons in <think> tags and outputs a sequence of tool calls in <tool> tags. Three tools are available—captioning (question-conditioned dense caption via a frozen VLM-base), grounding (bounding-box crop of the focal object via VLM-base), and flipping (horizontal image mirror to reduce orientation bias). Outputs are aggregated into a multimodal retrieval query Query = {C_init, I_grounding}.

  2. Multimodal retrieval stage: A frozen EVA-CLIP 8B embeds the query; Faiss with cosine similarity retrieves the top-1 article from direct image search plus top-5 articles via tool-generated queries from a million-scale Wikipedia KB. Articles are split into sections; top-k sections are re-ranked per-tool using cosine similarity between the tool output embedding and section embeddings.

  3. Filtering stage: VLM-PRF receives the raw retrieved sections and article text, reasons in <think> tags, then outputs a compact, question-focused knowledge summary F in <answer> tags. F is passed to a frozen base VLM for final answer generation.

RL training uses GRPO with a composite reward: answer accuracy (exact match, weight α=1.0) + tool format compliance (regex, β=0.3) + filtering format compliance (regex, γ=0.7). Only 4K training samples are needed; training takes ~15 hours on 8×A800 GPUs.

Key Contributions

  • Three-stage PRF framework for KB-VQA combining dynamic visual tool invocation, multimodal section retrieval, and question-conditioned filtering.
  • VLM-PRF: first application of RL (GRPO) to train a VLM for multimodal RAG; achieves strong generalization with only 4K training examples vs. 934K–1M for supervised baselines.
  • State-of-the-art on E-VQA (36.0 BEM) and InfoSeek (42.8 VQA accuracy), plus OK-VQA (77.8).
  • Demonstrated robustness to knowledge base scale (slower accuracy degradation as KB grows from 10K→100K vs. vanilla multimodal RAG).

Results

  • E-VQA (BEM score, test set): VLM-PRF-7B: 36.0 vs. ReflectiVA 35.9, MMKB-RAG 28.6, EchoSight 29.2, Qwen2.5-VL-7B base 20.3.
  • InfoSeek (VQA accuracy, validation): VLM-PRF-7B: 42.8 vs. ReflectiVA 40.0, MMKB-RAG 39.5 (w/o RL), Qwen2.5-VL-7B base 23.7.
  • OK-VQA: Wiki-PRF-7B: 77.8 vs. KU-RAG 73.1, MMKB-RAG 65.4, Qwen2.5-VL-7B base 72.4.
  • Retrieval recall (InfoSeek): Image-only 45.56% → image + tools (7B) 53.44% → with RL (VLM-PRF-7B) 54.89%.
  • Oracle filtering (E-VQA): Wiki-PRF-7B 65.8% vs. ReflectiVA 57.6%, Wiki-LLaVA 51.5%.
  • RL vs. SFT (InfoSeek, 2K samples): RL 46.3% vs. SFT 41.8% vs. base 40.2%.
  • Per-stage ablation (InfoSeek 10K): processing stage alone +2.54%, filtering stage alone +2.02%, both combined +5.26% over baseline.

Limitations

  • Tool set is fixed to three operations (captioning, grounding, flipping); no tool for OCR, depth, or external search APIs.
  • Processing and filtering stages add significant latency (6.2s/sample for 3B, 8.8s/sample for 7B), limiting real-time applicability.
  • Retrieval still degrades as knowledge base scales (10K→100K), a universal RAG challenge not fully resolved.
  • Evaluation is restricted to entity-centric Wikipedia-grounded datasets; generalization to other KB types (e.g., scientific literature, structured databases) is untested.
  • The example in Figure 3 shows VLM-PRF still producing incorrect answers (ground truth "Canada" vs. predicted "United States and parts of Canada"), indicating imperfect filtering.

Relevance to Vision-Language Models

This work directly advances how VLMs can be augmented with external knowledge at inference time—a core open problem for deploying VLMs on factual, entity-centric queries that exceed parametric knowledge. The RL-based training paradigm for tool-use and retrieval filtering is particularly relevant: it shows that GRPO with answer-accuracy reward can teach a 3B–7B VLM to adaptively orchestrate retrieval tools without large supervised datasets, complementing the broader trend of using RL to improve VLM reasoning (e.g., R1-style models). The processing stage's grounding-before-retrieval design also connects to compositional attention and fine-grained visual understanding challenges central to VLM alignment research.