Skip to content

Physics Question Scene Graph: Fine-grained Evaluation of Physical Plausibility in Text-to-Video Generation

🕒 Published (v1): 2026-06-24 02:12 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

Bansal group (UNC, ECCV 2026); scene-graph eval for physical plausibility in T2V — new rigorous eval tool

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PQSG (Physics Question Scene Graph) is a hierarchical, question-based evaluation framework that uses VLMs to generate and answer structured DAG queries decomposed into Object, Action, and Physics categories for assessing physical plausibility in text-to-video generation. It outperforms all prior metrics in correlation with human judgments and provides fine-grained localization of failure modes. The accompanying FinePhyEval dataset covers 195 human-annotated videos from Sora 2, Veo 3, and Wan 2.1.

Problem

Existing text-to-video evaluation metrics (VideoScore, VideoPhy-2, PhyGenEval) produce coarse aggregate scores that cannot localize which physical law is violated or why a video fails. They conflate object existence, action correctness, and physics plausibility into a single number, making them easily fooled by visually realistic but physically implausible videos and providing no actionable feedback for model improvement or fine-grained video repair.

Method

PQSG operates as a two-stage VLM pipeline over a directed acyclic graph (DAG):

  1. Question Generation (QG): A VLM (Gemini-2.5-Pro or GPT-5.5) receives the text prompt and one in-context example, then generates atomic yes/no verification questions organized into three node types — Object (entity existence), Action (prompt-specified dynamics), and Physics (implicit physical commonsense, e.g., fluid absorption, deformation) — connected by explicit dependency edges.

  2. Question Answering (QA): A VLM is given the generated video and each PQSG node question sequentially. Answers are produced in two sub-steps: open-ended chain-of-thought response, then binary yes/no categorization using the same model. Dependency enforcement is strict: if a parent node is answered "no," all descendant nodes are automatically marked "no" without querying, preventing hallucinated physics judgments on nonexistent objects or actions.

Scores are computed as ratios of "yes" responses per category (Object, Action, Physics) and aggregated into an overall score.

Key Contributions

  • PQSG framework: hierarchical DAG-based physics evaluation with explicit logical dependencies between Object → Action → Physics nodes, separating generation failures from physical implausibility failures.
  • FinePhyEval dataset: 195 human-annotated prompt-video pairs (65 prompts × 3 models: Sora 2, Veo 3, Wan 2.1) with four-category Likert scores (object, action, physics, overall; ICC = 0.84), plus 444 manually annotated QA pairs and ground-truth QG annotations for 20 prompts.
  • VLM subtask benchmarking: evaluation of Gemini-2.5-Pro and GPT-5.5 on QG precision/recall and QA accuracy against human ground truth, exposing a significant VLM gap on QA (~65% accuracy for GPT-5.5).
  • Generalization: demonstrated on an external dataset and with an open-source VLM.

Results

  • Correlation with human overall Likert scores (Pearson's \(r\) / Kendall's \(\tau\) / Spearman's \(\rho\)):
  • PQSG w/ GPT-5.5: 0.478 / 0.336 / 0.456 — highest across all metrics
  • PQSG w/ Gemini-2.5-Pro: 0.467 / 0.306 / 0.406
  • Direct VQA (Gemini-2.5-Pro): 0.382 / 0.290 / 0.360
  • VideoPhy-2-Autoeval: 0.346 / 0.277 / 0.349
  • VideoScore: 0.289 / 0.262 / 0.378
  • DSG: 0.302 / 0.195 / 0.265
  • PhyGenEval: 0.272 / 0.220 / 0.264
  • Model ranking by PQSG: proprietary models (Sora 2, Veo 3) outperform open-source (Wan 2.1, Cosmos-Predict2.5-14B); all models struggle more on Action and Physics than Object.
  • QA accuracy: best model GPT-5.5 achieves ~65% on action/physics QA; human upper-bound QA shows substantially higher correlation, indicating room for improvement as VLMs advance.
  • Dependency graph ablation (Table 8): dependency structure yields stronger human correlation than unstructured question sets.
  • Inter-annotator agreement: ICC = 0.84 across four Likert categories.

Limitations

  • QA accuracy is bottlenecked by current VLM capability (~65% for best model), making automatic PQSG scores a lower bound relative to human-QA PQSG.
  • Prompts sourced exclusively from Physics-IQ (65 prompts); domain and compositional diversity of evaluation set is limited.
  • Videos are short (avg. 4.39 s), so long-horizon physical dynamics are not captured.
  • QG relies on one in-context example; quality of generated questions depends on VLM instruction following and may miss physics phenomena not salient in the prompt.
  • Cosmos-Predict2.5-14B, despite physics-simulator architecture, does not achieve markedly better results, but analysis of why is limited.
  • No evaluation of PQSG sensitivity to prompt ambiguity or edge cases where Object/Action/Physics categories overlap.

Relevance to Vision-Language Models

PQSG directly benchmarks frontier VLMs (Gemini-2.5-Pro, GPT-5.5) in two structured reasoning roles — question graph generation and video-grounded binary QA — exposing a specific gap: VLMs are adequate at generating human-like evaluation questions but fall well short of human performance when answering them about physical dynamics in short videos. This is a concrete failure mode for VLMs used as evaluators or world-model judges, which is increasingly central to the VLM-as-judge and video-LMM literature. The DAG-with-dependency-enforcement design also offers a reusable pattern for grounding VLM evaluation in logical preconditions, applicable beyond physics to any structured video understanding task.