DeforM: Reasoning-Guided Physics-Aware Video Generation via Spatial-Temporal Masking

🕒 Published (v1): 2026-07-21 03:23 UTC · Source: Arxiv · link

Why this paper was selected

Yu Qiao (Shanghai AI Lab); reasoning-guided physics-aware video generation via spatial-temporal masking

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DeforM is an image-to-video generation framework that uses a VLM as a physical reasoning agent to localize deformation-critical regions and generate binary spatial-temporal masks, then injects those masks into a diffusion transformer to produce physically plausible deformation dynamics (melting, squeezing, fracturing, etc.). It offers two variants: a training-free attention modulation method (DeforM-Free) and a training-based dense mask injection method (DeforM-Injection). DeforM-Injection outperforms strong I2V baselines on both visual quality and VLM-graded physical consistency metrics.

Problem

Existing video generation models lack physical reasoning: they cannot autonomously associate text-described deformation phenomena (e.g., "melting") with the correct spatial pixels in the conditioning image, causing attention to dilute across irrelevant regions and producing physically implausible outputs. Prior VLM-guided approaches (VLIPP, PhyRPR) are limited to rigid-body motion via trajectory bounding boxes and cannot capture high-dimensional deformation dynamics like fracturing or stretching.

Method

DeforM-Reason (VLM-guided localization): Given image \(c_{img}\) and text \(c_{text}\), Qwen2.5-VL-7B identifies the target object \(O_{target}\) most relevant to the described deformation. An open-vocabulary detector (LLMDet) grounds \(O_{target}\) in the conditioning frame to produce \(B_1\); for inference, the VLM temporally extrapolates a bounding-box sequence \(\{B_i\}_{i=2}^{T}\). These boxes form a binary spatial-temporal mask \(M \in \{0,1\}^{T \times H \times W}\), downsampled to latent dimensions \((T', H', W')\).

DeforM-Free (training-free): Cross-attention queries in the DiT backbone are scaled within masked tokens: $\(\tilde{\mathbf{Q}}_j = \mathbf{Q}_j \cdot [1 + \hat{\mathbf{M}}_j \cdot (\alpha - 1)], \quad \alpha = 1.4\)$ This amplifies attention scores on physics-critical regions without modifying weights.

DeforM-Injection (training-based): A Dense Mask Modulator \(F_\phi\) (3D convolutions + zero-initialized linear layers) encodes \(M\) into three offset streams \(\Delta k_{self}, \Delta v_{self}, \Delta q_{cross}\), injected additively into self-attention keys/values and cross-attention queries respectively. LoRA (rank 256) is co-trained on the backbone. The training loss combines global flow matching loss with a region-sensitive mask loss: $\(\mathcal{L}_{total} = (1 - \lambda_{mask})\mathcal{L}_{flow} + \lambda_{mask}\mathcal{L}_{mask}, \quad \lambda_{mask} = 0.3\)$ The base I2V model is Wan2.2-TI2V-5B; training uses ~6k high-quality deformation samples from WISA-80K.

Key Contributions

• DeforM-Reason: VLM+open-vocabulary detector pipeline that converts abstract physical text prompts into explicit spatial-temporal masks aligned with video latents.
• DeforM-Free: plug-and-play training-free cross-attention query modulation that empirically validates the semantic-spatial gap hypothesis.
• DeforM-Injection: dense mask modulator with synchronized self- and cross-attention injection plus region-sensitive loss, trained with LoRA for diverse deformation realism.
• A three-dimensional physical consistency evaluation protocol (Physical Commonsense, Semantic Adherence, Local Deformation Fidelity) scored by Qwen3-VL-32B.

Results

Evaluated on 320-sample test set from WISA-80K; physical consistency on 160 samples; baselines: CogVideoX1.5-I2V, Hunyuan-I2V, Wan2.2-I2V-14B, MAGI-1.

• Overall Quality Score: DeforM-Injection 73.17 vs. best baseline Wan2.2-14B 71.18 (+~2 points).
• Dynamic Degree: DeforM-Injection 54.74 vs. best baseline CogVideoX1.5 48.32 (+~6 points); Hunyuan-I2V scores only 27.74, indicating near-static generation.
• Physical Score (mean of PC/SA/LDF): DeforM-Injection 4.40 vs. best baseline Wan2.2-14B 3.95 (+0.45, ~+11%).
• Local Deformation Fidelity: DeforM-Injection 4.16 vs. best baseline Wan2.2-14B 3.65 (+0.51, ~+14%).
• Physical Commonsense: DeforM-Injection 4.59 vs. best baseline Wan2.2-14B 4.28 (+0.31, ~+7%).
• DeforM-Injection trades slight temporal flickering and motion smoothness scores vs. Hunyuan-I2V (97.24/98.40 vs. 99.30/99.54) in exchange for substantially higher dynamic degree and physical fidelity.

Limitations

• Inference-time spatial-temporal mask relies on VLM temporal extrapolation from a single frame, which may be inaccurate for complex or unpredictable deformation trajectories.
• Training data is limited to ~6k samples from WISA-80K covering specific deformation categories; generalization to out-of-distribution deformation types is unproven.
• DeforM-Free is bounded by the base model's pre-existing knowledge; if the base model lacks an internal representation of a deformation type, attention scaling alone produces artifacts.
• Evaluation is performed by a VLM (Qwen3-VL-32B) rather than human raters or physics simulators, introducing potential model-specific biases.
• Subject consistency and background consistency metrics are excluded from VBench evaluation by design, which limits comparability with standard benchmarks.

Relevance to Vision-Language Models

DeforM is a direct application of VLMs as grounded physical reasoning agents: Qwen2.5-VL-7B performs both object identification from multimodal input and temporal planning, serving as the backbone for spatial-temporal mask generation rather than as a purely evaluative component. This demonstrates a concrete instantiation of VLMs beyond classification or captioning — as structured spatial planners that condition downstream generative models. For VLM researchers, the result that a 7B VLM can reliably localize deformation regions and extrapolate bounding-box sequences well enough to improve physical consistency by ~11% over 5B–14B pure video generators is a strong signal about VLMs' emergent spatial-temporal reasoning capabilities. The framework also uses a 32B VLM (Qwen3-VL) as a physics-aware evaluation judge, advancing the paradigm of VLM-as-evaluator for perceptual quality beyond standard NLP tasks.