Skip to content

PhysAgent: Reflective Agentic Physics Control for Physically Plausible Video Generation

🕒 Published (v1): 2026-07-17 08:55 UTC · Source: Arxiv · link

Why this paper was selected

Reflective agentic physics control for video gen; VLM+agent+simulation pipeline

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PhysAgent is a closed-loop agentic framework that replaces single-pass VLM prediction of physics simulation parameters with iterative execution-verify-repair cycles. Given a single image and a physical event prompt, a VLM generator, a stage-specific verifier, and a planner jointly refine a symbolic physical control program until the simulated dynamics satisfy the intended event. This yields substantially better physical plausibility and prompt alignment than prior physics-grounded video generation methods.

Problem

Physics-grounded video generation methods rely on a VLM to predict a complete simulator specification (object positions, materials, forces, timing) in one shot. Because all these parameters are tightly coupled, a locally plausible estimate can still produce a rollout that fails the intended event — objects slide without contact, forces act in the wrong direction, or material responses are ignored. Single-pass prediction cannot self-correct because specification correctness is only observable through execution.

Method

PhysAgent represents the simulation configuration as a physical control program \(P_k = (S_k, E_k)\), where \(S_k\) is the scene state (object coordinates, materials, densities, camera) and \(E_k\) is an event program expressed through a compact Action API (SetVelocity, ApplyForce, ApplyTorque, ApplyAngledForce, ApplyDisturbance, SetPosition, SetOrientation, FixObject). The reflective loop has three roles:

  1. Generator \(G\): constructs \(P_1 = G(I, p)\) from the input image \(I\) and event prompt \(p\); at round \(k>1\), revises via \(P_k = G(P_{k-1}, F_{k-1})\).
  2. Verifier \(V\): applies progressive validation — first checks a rendered view of the reconstructed 3D scene; if that passes, executes the event program and checks a subsampled simulated video \(\tilde{X}_k = \{x^k_1, x^k_4, x^k_7, \ldots, x^k_{79}\}\) against the event prompt for trajectory, contact timing, force direction, and material response. Outputs a pass/fail decision plus a concise diagnosis \(A_k\).
  3. Planner \(P\): converts the diagnosis \(A_k\) and a cumulative memory \(M_k\) of prior edits into a structured repair proposal \(F_k = P(A_k, M_k)\), targeting only the failed factor without disturbing already-validated decisions.

A verified rollout's optical flow is passed as a motion prior to a downstream video diffusion model. The maximum reflection budget is 10 rounds with early exit on passing verification.

Key Contributions

  • Reformulates physics-grounded video generation as execution-aware hypothesis testing: each physical program is an executable candidate, not a one-shot prediction.
  • Reflective agentic loop closing the cycle between VLM-based program generation, physics simulation, stage-specific verification, and targeted parameter repair.
  • Semantic physics-control API abstracting brittle low-level simulator commands into motion-level primitives (staged forces, disturbances, fixation/release, pose constraints).
  • A new physical-event benchmark of 27 scenes × 2–4 event descriptions (80 cases total) with same-scene multi-event design to isolate physical controllability from scene appearance.
  • PhysReal: a GPT-based normalized physical plausibility + prompt-alignment metric evaluated per video frame sequence.

Results

  • PhysReal (physical plausibility, higher is better): PhysAgent 0.714 vs. RealWonder 0.631, Wan2.2 14B 0.633, Goal-Force 0.671, PhysCtrl 0.579, PhysGen3D 0.480.
  • Aesthetics: PhysAgent 0.521 (best among all compared methods).
  • Visuals / Consistency: 0.644 / 0.266 (competitive but not dominant; text-to-video baselines with strong video priors remain close on perceptual metrics).
  • 2AFC user study (PhysAgent favor rate):
  • Over PhysGen3D: 92.4% action following, 90.3% visual quality, 93.6% physical plausibility.
  • Over PhysCtrl: 90.8% / 88.9% / 92.1%.
  • Over Wan2.2 14B: 66.8% / 60.7% / 73.4%.
  • Ablation table values not provided in the supplied text excerpt.

Limitations

  • Small benchmark: 27 scenes / 80 event cases; generalization to broader distributions is unverified.
  • Compute cost of reflection: up to 10 simulation rounds per sample; latency is not reported in the provided text.
  • Upstream dependency: scene reconstruction relies on SAM3D; reconstruction failures propagate to downstream simulation despite the staged validation guard.
  • GPT-based evaluation: PhysReal depends on GPT-5.5 as a judge, introducing potential model-specific biases into the primary physics metric.
  • Paper text truncated: the ablation study and limitations section were cut off, so stated limitations from the authors cannot be fully reported.

Relevance to Vision-Language Models

PhysAgent is directly a VLM-use-case paper: the generator, verifier, and planner are all VLM calls, and the core contribution is a prompting + execution architecture that compensates for VLM single-pass brittleness on structured numerical prediction tasks. It demonstrates that wrapping a VLM in a reflection loop with tool-executable programs and stage-specific visual feedback — rather than one-shot generation — substantially improves reliability on tasks requiring precise parameter calibration. For VLM researchers, this is a concrete instantiation of code-as-thought / executable-hypothesis reasoning in a physics domain, with a rigorously constructed benchmark showing where standard VLM inference fails. It connects to the broader line of work on VLMs as agents operating in executable environments (ViperGPT, BlenderGym, SceneCraft), extending that line to temporally structured physical simulation.