Skip to content

VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VRAG-RL is a reinforcement learning framework that trains VLMs to iteratively retrieve and reason over visually rich document collections using a visual perception action space (region crop/zoom) and a composite RAG-specific reward. Applied to Qwen2.5-VL-3B and 7B, it achieves 53.5% and 57.1% overall accuracy on multi-benchmark evaluation, outperforming the strongest prior visual RAG baseline (Search-R1-VL) by ~32 and ~20 points respectively.

Problem

Traditional text-based RAG cannot handle visual information in documents (charts, layouts, slides). Existing vision-based RAG methods suffer from three compounding failures: (1) VLMs are given retrieved images as flat context without any visual-specific perception, leading to insufficient reasoning-token allocation; (2) search queries fail to retrieve relevant images because models cannot articulate complex visual requirements; (3) multi-turn RL training with VLMs is unstable and inconsistent across interaction steps.

Method

VRAG-RL fine-tunes VLMs (Qwen2.5-VL-3B/7B) via GRPO on multi-turn trajectories that interleave three action types: <search> (query a visual document store of ~70k images), <region> (select a bounding box, crop the raw image at native resolution, and re-inject it into context as a higher-density visual observation), and <answer>. The visual perception tokens enable a coarse-to-fine inspection loop: the model can retrieve a full-page image, then zoom into a data-dense sub-region without re-querying the search engine.

The reward combines three terms (weighted α·r_Ret + β·r_Ans + γ·r_Pat): - Retrieval efficiency reward (r_Ret): modified NDCG over the stacked retrieved images across all turns, rewarding early and complete retrieval of relevant images. - Model-based outcome reward (r_Ans): an LLM judge evaluates the final answer against a gold reference, replacing brittle EM or recall-based rewards. - Pattern reward (r_Pat): rule-based check that actions follow the predefined action-space format; weight γ=0.1 only during cold-start RL, otherwise 0.

Training proceeds in two stages: (1) SFT on trajectories generated via multi-expert sampling (large model πLM guides reasoning; smaller grounding model πEM refines bounding-box coordinates); (2) GRPO RL with group size 5 and KL coefficient 0.01.

Key Contributions

  • Visual perception action space (select→crop→re-encode) integrated into the VLM's Thought-Action-Observation loop, enabling dynamic resolution enhancement within fixed encoder pixel budgets.
  • NDCG-based retrieval efficiency reward that explicitly penalizes late or absent retrieval of relevant images across the full multi-turn trajectory.
  • Model-based outcome reward that avoids EM brittleness and recall-hacking failure modes during RL training.
  • Multi-expert sampling pipeline for scaling high-quality SFT trajectory data: large model for reasoning paths, specialist grounding model for bounding-box precision.
  • Demonstration that RL reduces invalid action rate from 9.4% to 5.1% and raises trajectory finish rate from 84.2% to 97.1%.

Results

  • Overall accuracy (Qwen2.5-VL-7B): VRAG-RL 57.1% vs. Search-R1-VL 37.4% (+19.7 pp); vs. Vanilla RAG visual 24.2%.
  • Overall accuracy (Qwen2.5-VL-3B): VRAG-RL 53.5% vs. Search-R1-VL 21.3% (+32.2 pp); vs. Vanilla RAG visual 13.2%.
  • SlideVQA single-hop (7B): 69.3% vs. Search-R1-VL 48.3%; (3B): 65.3% vs. 26.3%.
  • ViDoSeek Logic (7B): 74.8% vs. 50.3%; (3B): 73.8% vs. 29.8%.
  • MMLongBench Chart (7B): 26.3% vs. 13.4%; Layout (7B): 25.9% vs. 11.4%.
  • Ablation (7B): removing visual-perception action space drops overall from 57.1→54.9; removing RAG-specific retrieval reward drops to 49.3; removing both drops to 47.2.
  • VRAG-RL achieves higher Recall@1 than ReAct and direct-retrieval baselines across all three benchmarks.

Limitations

  • Increased inference latency due to multi-turn interactions with the search engine and iterative crop-and-re-encode calls.
  • Requires an annotated image corpus (~70k documents) with known relevance labels to compute the NDCG retrieval reward; not directly applicable to fully unsupervised retrieval corpora.
  • Multi-expert SFT pipeline is complex: requires a large frontier VLM for trajectory generation and a separate grounding model for coordinate annotation.
  • Evaluated only on Qwen2.5-VL-3B and 7B; scalability to larger models or other VLM families is not demonstrated.
  • Model-based outcome reward introduces dependency on an external judge model, adding cost and a potential evaluation bottleneck.

Relevance to Vision-Language Models

VRAG-RL directly advances VLM capability in agentic, tool-augmented settings by showing that RL-tuned visual perception actions—not just better retrieval prompts—are necessary for handling information-dense documents. It demonstrates that the standard practice of inserting full retrieved images into context is a fundamental bottleneck, and that crop-and-re-encode loops can substitute for prohibitively high-resolution encoders. The NDCG-based retrieval reward is a transferable design pattern for any VLM RAG pipeline where retrieval quality is a latent variable. Together with concurrent work (Search-R1, VisionR1), this paper establishes RL with environment interaction as a core training paradigm for next-generation VLMs on document understanding tasks.