Skip to content

PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Current MLLMs perform visual reasoning only on static, fully observable images, failing when information is occluded or out of view. This paper introduces Active Visual Reasoning (AVR), a closed-loop paradigm requiring agents to sequentially take physical actions to gather missing information, reason over accumulated observations, and dynamically update decisions. The authors contribute a benchmark (CLEVR-AVR), a 152k training dataset with structured Chain-of-Thought annotations (AVR-152k), and a trained model (PhysVLM-AVR-3B) achieving state-of-the-art on the new benchmark.

Problem

Visual reasoning benchmarks (CLEVR, VQA) assume complete, static observations; embodied QA tasks (OpenEQA, RoboVQA) allow passive video observation but do not require goal-directed information gathering. No existing framework closes the loop between reasoning about what is unknown, selecting actions to resolve that uncertainty, and integrating new observations—making current MLLMs brittle in partially observable physical environments where objects are occluded or hidden in containers.

Method

AVR is formalized as a higher-order Markov Decision Process: at each timestep \(t\), the agent receives observation history \(h_t\), reasons about sufficiency, and either answers or selects action \(a_t = \arg\max_{a \in A} \mathbb{E}[I(Y; y_{t+1} | h_{t+1}, Q)]\) to maximize expected information gain about the true answer.

CLEVR-AVR benchmark extends CLEVR via the Genesis physics simulator into an interactive domain with 10 occlusion types, 10 stacking types, and 10 composite scene types. Action candidates are mixed with final answer options, forcing explicit sufficiency judgment. Three metrics: Information Sufficiency Judgment Accuracy (ACC\(_{ISJ}\)), Information Gain Rate (IGR), and Final Answer Accuracy (ACC\(_{FA}\)).

AVR-152k dataset has three subsets of increasing complexity: - AVR-Caption (100k): dense captions + bounding boxes from ScanNet/RT1 generated by Gemini-2.0-flash. - AVR-Embodied Reasoning (50k): multi-image sequences with spatiotemporal QA; reasoning chains from DeepSeek-R1-671B refined by DeepSeek-V3. - AVR-Core (2k): real tabletop scenes collected with UMI devices; expert-authored CoTs covering (i) uncertainty hypothesis, (ii) action-conditioned information gain prediction per candidate action, (iii) strategic action/answer decision; verified by Gemini + human review.

PhysVLM-AVR uses Qwen2.5-3B as LLM decoder and SigLIP-400M as visual encoder, with a 3Ă— max-pooling compression on visual tokens for multi-image efficiency. Training is four-stage: (1) connector alignment on LLaVA-Pretrain; (2.1) single-image fine-tuning on LLaVA-OneVision-data; (2.2) broader understanding with M4-Instruct + AVR-Caption; (3) reasoning fine-tuning on Reason-RFT-129k, AM-DeepSeek-R1-Distilled-100k, AVR-Embodied Reasoning, and AVR-Core.

Key Contributions

  • Formal definition of the Active Visual Reasoning (AVR) task as a higher-order MDP covering active information acquisition, temporal integration, and dynamic decision-making.
  • CLEVR-AVR: simulation benchmark with three scene categories and three complementary metrics (ACC\(_{ISJ}\), IGR, ACC\(_{FA}\)).
  • AVR-152k: large-scale dataset with structured CoT annotations explicitly supervising uncertainty identification, action-conditioned information gain prediction, and action selection.
  • PhysVLM-AVR-3B: 3B-parameter model achieving state-of-the-art on CLEVR-AVR and competitive performance on OpenEQA, RoboVQA, GeoMath, and Geometry3K.
  • Diagnostic finding: existing embodied MLLMs can detect information incompleteness but fail to act strategically to resolve it.

Results

CLEVR-AVR (AVG across scene types): - PhysVLM-AVR-3B: ACC\(_{ISJ}\) 90.5%, IGR 29.9%, ACC\(_{FA}\) 39.7% - AVR-Qwen2.5-VL-7B: ACC\(_{ISJ}\) 89.3%, IGR 34.7%, ACC\(_{FA}\) 38.1% - GPT-4o: ACC\(_{ISJ}\) 88.4%, IGR 50.8%, ACC\(_{FA}\) 45.7% - Embodied-Reasoner-7B: ACC\(_{ISJ}\) 20.2%, IGR 10.9%, ACC\(_{FA}\) 1.6% - LLaVA-OV-7B / Qwen2.5-VL-7B: near-zero across all metrics

OpenEQA: PhysVLM-AVR-3B and AVR-Qwen2.5-VL-7B outperform standard MLLMs and dedicated embodied models (RoboBrain-7B) across all sub-tasks.

RoboVQA: Models trained on only 1/20 of the training set match or exceed fully supervised baselines on BLEU1–4.

GeoMath / Geometry3K: PhysVLM-AVR-3B surpasses GPT-4o and Reason-RFT-7B.

Ablations on CLEVR-AVR (PhysVLM-AVR-3B): - Removing AVR-Core: ACC\(_{ISJ}\) drops 90.5%→16.4%, ACC\(_{FA}\) drops 39.7%→2.3% - Removing CoT only: ACC\(_{ISJ}\) drops to 47.6%, ACC\(_{FA}\) drops to 16.9%

Limitations

  • AVR-Core contains only 2k real-world samples; the gap between 90.5% ACC\(_{ISJ}\) and 39.7% ACC\(_{FA}\) shows action selection and multi-step integration remain inadequate.
  • IGR for trained models (29.9–34.7%) remains well below GPT-4o (50.8%), indicating suboptimal information-gathering efficiency.
  • CLEVR-AVR is simulation-based (Genesis); generalization to uncontrolled real-world scenes is undemonstrated.
  • Action space is discrete and tabletop-scoped; scaling to richer manipulation or navigation spaces is unaddressed.
  • No reinforcement learning from environment feedback; training is fully supervised on CoT annotations.

Relevance to Vision-Language Models

This work directly challenges the static-observation assumption underlying virtually all VLM benchmarks, pushing MLLMs toward interactive, partially observable settings that better reflect real deployment. The higher-order MDP formulation and CoT-annotated dataset provide a principled recipe for teaching VLMs when and how to seek additional visual information—a capability absent from standard instruction-tuning pipelines. The diagnostic result—that embodied MLLMs can recognize incompleteness but cannot act to resolve it—identifies a concrete bottleneck for the community working on embodied VLMs, robotic agents, and agentic multimodal systems. CLEVR-AVR and AVR-152k together constitute a reusable infrastructure for benchmarking and training the next generation of actively-reasoning VLMs.