Skip to content

PhysMRV: Physical Memory Retrieval and Verification for Physics Plausibility Reasoning

🕒 Published (v1): 2026-07-11 08:06 UTC · Source: Arxiv · link

Why this paper was selected

Physical plausibility reasoning in video-LMs; targets a known VLM failure mode with retrieval+verification

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PhysMRV is a training-free retrieval-augmented framework that enhances frozen video-language models (VLMs) for physical plausibility reasoning by constructing a structured Hierarchical Physical Memory Bank from training videos and retrieving physically analogous memories at inference time. Unlike semantic retrieval baselines, it matches on physical-event graph structure to find mechanistically similar precedents, then uses their distilled rules and diagnostic cues to guide a frozen VLM in verifying whether observed events satisfy physical constraints. It achieves consistent accuracy gains on ImplausiBench, IntPhys2, and GRASP Level 2 across four VLM backbones without any parameter updates.

Problem

VLMs trained on large-scale video-language data develop strong physical plausibility priors that bias predictions toward physically reasonable interpretations regardless of actual visual evidence, causing them to fail at detecting violations of basic physical laws (e.g., unsupported objects, object non-permanence, causally inconsistent collisions). Existing retrieval-augmented approaches rely on semantic or surface-level similarity, which is insufficient because visually/linguistically similar events can differ in underlying physical constraints (support, contact, causality). Training-based fixes (e.g., TRAVL, PhysVLM) require parameter updates that preclude use with frozen or proprietary models.

Method

PhysMRV operates in three offline/online stages:

Offline — Hierarchical Physical Memory Bank construction from 3,482 TRAVL training videos (19,708 QA pairs). Each memory entry \(m_j\) contains: - Caption layer: coarse scene description (Qwen3-VL-8B, shared per video) - Physical-Event layer: structured event graph \(g_j^{\text{train}}\) with tuples \(({\rm event\_type, objects, state, evidence, phase})\), extracted by Qwen3-VL-8B - Rule layer: rule family (one of 15 categories), physical principle, positive cues \(C_j^+\), negative cues \(C_j^-\), generated via GPT-5.5 from QA explanations

Online — Coarse-to-fine retrieval: 1. Semantic recall: concatenate eval caption + question \(\tau_i^{\text{eval}}\); encode with Qwen3-Embedding-0.6B; cosine similarity retrieves top-50 candidates \(C_i^{50}\) 2. Physical-event reranking: for each candidate, compute graph-level score $\(S_{\text{graph}}(g_i^{\text{eval}}, g_j^{\text{train}}) = \frac{1}{|E_i^{\text{eval}}|}\sum_{e \in E_i^{\text{eval}}} \max_{e' \in E_j^{\text{train}}} \sum_{p \in P} \alpha_p\, u_p(e)^\top u_p(e')\)$ where \(\alpha_p = 0.2\) for each of the five event components. Rerank and greedily select top-3 anchors \(R_i\) from distinct rule families for diversity.

Online — Rule-cue verification: expose compact verification records \(V_j = (q_j^{\text{train}}, e_j, r_j, C_j^+, C_j^-)\) to a frozen VLM alongside the eval caption, eval event graph, and question via structured prompt \(P_i = \rho(q_i, A_i, \text{cap}_i^{\text{eval}}, g_i^{\text{eval}}, V_i)\). The VLM acts as a visual verifier comparing current evidence against retrieved physical principles and cues.

Key Contributions

  • Hierarchical Physical Memory Bank: three-level representation (caption / event graph / physics rule) organized under a unified 15-family physical rule taxonomy covering support, force, collision, continuity, fluid, thermal, optical, and causal phenomena
  • Physics-aware coarse-to-fine retrieval: semantic recall for high recall followed by physical-event graph alignment for physical relevance and rule-family diversity enforcement
  • Training-free plug-and-play framework: no fine-tuning or parameter updates to any backbone VLM; compatible with frozen and proprietary models

Results

All numbers are accuracy on the respective benchmark's native metric (option accuracy for ImplausiBench; balanced possible/impossible accuracy for IntPhys2; binary accuracy for GRASP Level 2). Baseline is zero-shot direct prompting with the same VLM.

  • Qwen3-VL: ImplausiBench 0.550 → 0.590 (+4.0 pp); IntPhys2 0.499 → 0.540 (+4.1 pp); GRASP L2 0.484 → 0.580 (+9.6 pp)
  • Cosmos-Reason2: ImplausiBench 0.533 → 0.573 (+4.0 pp); IntPhys2 0.501 → 0.537 (+3.6 pp); GRASP L2 0.500 → 0.566 (+6.6 pp)
  • InternVL3.5-8B: ImplausiBench 0.543 → 0.583 (+4.0 pp); IntPhys2 0.499 → 0.535 (+3.6 pp); GRASP L2 0.497 → 0.572 (+7.5 pp)
  • LLaVA-OneVision-2: ImplausiBench 0.340 → 0.357 (+1.7 pp); IntPhys2 0.480 → 0.502 (+2.2 pp); GRASP L2 0.489 → 0.508 (+1.9 pp)
  • Ablations show that physical-event graph reranking outperforms caption-only retrieval (+Cap.) and graph-only ablations, and that rule-cue verification adds value over memory retrieval alone (Mem.)

Limitations

  • Gains on LLaVA-OneVision-2 are substantially smaller than on Qwen3-VL/Cosmos/InternVL, suggesting backbone capability is a ceiling
  • Memory Bank is built exclusively from TRAVL training data; cross-dataset generalization of the physical rule taxonomy is not empirically validated beyond the three benchmarks
  • Rule and cue generation relies on GPT-5.5 API, introducing cost and proprietary dependency in the offline construction phase
  • Physical-event graphs are extracted by a generative VLM (Qwen3-VL-8B) rather than a symbolic parser; extraction errors propagate to retrieval quality
  • Only tested on binary/multiple-choice plausibility tasks; applicability to generative physical reasoning is undemonstrated
  • The paper text is truncated before ablation analysis and qualitative failure cases, so the full scope of limitations is not available

Relevance to Vision-Language Models

PhysMRV directly targets a well-documented blind spot of VLMs—physical plausibility reasoning—and demonstrates that structured external memory can compensate for gaps that neither scale nor semantic retrieval alone addresses. The framework is model-agnostic and training-free, making it a practical inference-time augmentation layer for any frozen VLM, including proprietary ones. The 15-family rule taxonomy and hierarchical memory design offer a reusable scaffold for building physics-aware retrieval systems. For researchers tracking VLMs, the paper's finding that physical-event graph similarity outperforms semantic similarity as a retrieval signal is an actionable design principle for future RAG-style VLM augmentation work.