Skip to content

PhysAgent: Reflective Agentic Physics Control for Physically Plausible Video Generation

🕒 Published (v1): 2026-07-17 08:55 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PhysAgent is a closed-loop agentic framework that replaces single-pass VLM prediction of physics simulation parameters with iterative execution-verify-repair cycles. Given a single image and a physical event prompt, a VLM generator, a stage-specific verifier, and a planner jointly refine a symbolic physical control program until the simulated dynamics satisfy the intended event. This yields substantially better physical plausibility and prompt alignment than prior physics-grounded video generation methods.

Problem

Physics-grounded video generation methods rely on a VLM to predict a complete simulator specification (object positions, materials, forces, timing) in one shot. Because all these parameters are tightly coupled, a locally plausible estimate can still produce a rollout that fails the intended event — objects slide without contact, forces act in the wrong direction, or material responses are ignored. Single-pass prediction cannot self-correct because specification correctness is only observable through execution.

Method

PhysAgent represents the simulation configuration as a physical control program \(P_k = (S_k, E_k)\), where \(S_k\) is the scene state (object coordinates, materials, densities, camera) and \(E_k\) is an event program expressed through a compact Action API (SetVelocity, ApplyForce, ApplyTorque, ApplyAngledForce, ApplyDisturbance, SetPosition, SetOrientation, FixObject). The reflective loop has three roles:

  1. Generator \(G\): constructs \(P_1 = G(I, p)\) from the input image \(I\) and event prompt \(p\); at round \(k>1\), revises via \(P_k = G(P_{k-1}, F_{k-1})\).
  2. Verifier \(V\): applies progressive validation — first checks a rendered view of the reconstructed 3D scene; if that passes, executes the event program and checks a subsampled simulated video \(\tilde{X}_k = \{x^k_1, x^k_4, x^k_7, \ldots, x^k_{79}\}\) against the event prompt for trajectory, contact timing, force direction, and material response. Outputs a pass/fail decision plus a concise diagnosis \(A_k\).
  3. Planner \(P\): converts the diagnosis \(A_k\) and a cumulative memory \(M_k\) of prior edits into a structured repair proposal \(F_k = P(A_k, M_k)\), targeting only the failed factor without disturbing already-validated decisions.

A verified rollout's optical flow is passed as a motion prior to a downstream video diffusion model. The maximum reflection budget is 10 rounds with early exit on passing verification.

Key Contributions

  • Reformulates physics-grounded video generation as execution-aware hypothesis testing: each physical program is an executable candidate, not a one-shot prediction.
  • Reflective agentic loop closing the cycle between VLM-based program generation, physics simulation, stage-specific verification, and targeted parameter repair.
  • Semantic physics-control API abstracting brittle low-level simulator commands into motion-level primitives (staged forces, disturbances, fixation/release, pose constraints).
  • A new physical-event benchmark of 27 scenes × 2–4 event descriptions (80 cases total) with same-scene multi-event design to isolate physical controllability from scene appearance.
  • PhysReal: a GPT-based normalized physical plausibility + prompt-alignment metric evaluated per video frame sequence.

Results

  • PhysReal (physical plausibility, higher is better): PhysAgent 0.714 vs. RealWonder 0.631, Wan2.2 14B 0.633, Goal-Force 0.671, PhysCtrl 0.579, PhysGen3D 0.480.
  • Aesthetics: PhysAgent 0.521 (best among all compared methods).
  • Visuals / Consistency: 0.644 / 0.266 (competitive but not dominant; text-to-video baselines with strong video priors remain close on perceptual metrics).
  • 2AFC user study (PhysAgent favor rate):
  • Over PhysGen3D: 92.4% action following, 90.3% visual quality, 93.6% physical plausibility.
  • Over PhysCtrl: 90.8% / 88.9% / 92.1%.
  • Over Wan2.2 14B: 66.8% / 60.7% / 73.4%.
  • Ablation table values not provided in the supplied text excerpt.

Limitations

  • Small benchmark: 27 scenes / 80 event cases; generalization to broader distributions is unverified.
  • Compute cost of reflection: up to 10 simulation rounds per sample; latency is not reported in the provided text.
  • Upstream dependency: scene reconstruction relies on SAM3D; reconstruction failures propagate to downstream simulation despite the staged validation guard.
  • GPT-based evaluation: PhysReal depends on GPT-5.5 as a judge, introducing potential model-specific biases into the primary physics metric.
  • Paper text truncated: the ablation study and limitations section were cut off, so stated limitations from the authors cannot be fully reported.

Relevance to Harnesses / Meta-Harnesses

PhysAgent is a direct instantiation of the execute-observe-reflect harness pattern — its generate/verify/repair loop over symbolic physical control programs mirrors the scaffolding structure used in code-execution and tool-use agents, making it a concrete case study in how closed-loop agentic control outperforms single-pass prediction. The key transferable design is stage-specific verifiers: rather than one monolithic reward signal, each phase of the simulation pipeline gets a dedicated quality gate, which is precisely the kind of modular verification scaffolding that meta-harness designers need when orchestrating multi-step, multi-tool agents. If you're building harnesses that wrap generative models over structured simulation or environment state, the planner-verifier decomposition here gives you a reusable blueprint for when and how to trigger repair cycles without collapsing all quality signals into a single scalar. The physics domain is incidental — the architecture generalizes to any harness where outputs must satisfy compositional, domain-specific constraints that no single forward pass reliably enforces.