HAT-4D: Lifting Monocular Video for 4D Multi-Object Interactions via Human-Agent Collaboration¶

🕒 Published (v1): 2026-06-26 16:05 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV; monocular→4D pipeline for scalable VLA training data at low cost

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

HAT-4D is a human-agent collaborative agentic framework that lifts monocular video into physically plausible 4D multi-object interactions via an Interaction Knowledge Graph (IKG) built by VLMs and a multi-level human-in-the-loop (HITL) correction scheme. It addresses the gap between expensive multi-camera rigs and unreliable generative models for reconstructing real-world object interactions. The framework also serves as a data engine to construct MVOIK-4D, the first open-world benchmark for monocular 4D interaction reconstruction.

Problem¶

Existing monocular 4D reconstruction methods focus on isolated objects and fail under severe mutual occlusions, complex topological deformations, and multi-object dynamics. Hardware-intensive multi-camera systems are unscalable and lab-confined; generative approaches (SDS-based or dataset-driven) suffer domain gaps when applied to real-world interactions, producing interpenetration artifacts, floating objects, and temporal jitter. No open-world benchmark exists for evaluating physical plausibility and temporal consistency in monocular multi-object 4D reconstruction.

Method¶

Given an in-the-wild monocular video, HAT-4D proceeds in three stages:

IKG Construction. Qwen3-VL parses the video into a dynamic directed graph \(G = (O, E, R)\), encoding: object entities \(O\) with physical attributes (deformability, color, affordance); interaction event segments \(E\) partitioned by state transitions with flagged keyframes \(\ddot{T}_i\); and spatio-temporal relations \(R\) as tuples \(r_i = \langle (o_a, o_b), I, O_\text{depth}, S \rangle\) capturing semantic predicates, depth ordering \(O_\text{depth} \in \{\prec_\text{depth}, \succ_\text{depth}, \emptyset\}\), and 3-axis positional alignment \(S \in H \times V \times D\).
3D Generation and Composition. For each IKG keyframe, SAM3D segments and reconstructs per-object 3D Gaussian Splats. A composition skill then runs lightweight 6DoF pose optimization to place objects according to IKG spatial constraints and computes exact contact points for physical plausibility.
Memory-Augmented 4D Propagation. L4GM propagates the composed 3D scene segment-wise, conditioned on 4 rendered orthogonal planes and a memory bank (size 8) of previously refined keyframes to maintain long-term object identity. A dedicated evaluation agent checks each propagated segment against IKG constraints: dynamic assessment (interpenetration, memory stability) and static assessment (texture, cross-view consistency), triggering selective rollback—localized 4D re-propagation for physics errors, full rollback to 3D generation for geometry artifacts.

HITL Collaboration provides three levels of human refinement: Gaussian-level (direct splat attribute edits), region-level (local latent denoising via multi-view video generation), and object-level (full SAM3D re-generation with pose re-alignment). Human corrections are injected as online fine-tuning prompts to adapt agent behavior.

Key Contributions¶

HAT-4D: first HITL multi-agent framework for physically plausible and temporally coherent 4D multi-object interaction reconstruction from a single monocular video.
Interaction Knowledge Graph (IKG): a formal causality engine encoding object entities, event-segmented temporal structure, and physical relation tuples (depth ordering, contact constraints, interaction semantics) to guide 3D generation and 4D propagation.
MVOIK-4D benchmark: 77 tasks across 112 interaction scenarios, two subsets (ToolOIK-4D for complex deformations; MemoryOIK-4D for occlusion-driven spatio-temporal reasoning), with a multi-dimensional evaluation protocol covering CLIP/LPIPS/FVD, optical-flow-aligned intra-quality, DINOv3-based long-term memory similarity, and Qwen3-VL-judged interaction plausibility.
Scalable data engine: human-refined 4D assets serve as pseudo ground-truth for offline fine-tuning of 4D generation operators, bootstrapping physical priors for Embodied AI.

Results¶

HAT-4D achieves best LPIPS and FVD on MVOIK-4D across all baselines, indicating superior perceptual quality and temporal realism.
Baseline L4GM scores: CLIP 0.8259, LPIPS 0.1968, FVD 962.86, Deform 2.53, Relation 1.90, Intra 0.0013 (table truncated; full HAT-4D row not present in provided text).
Ablation: removing IKG degrades performance (replaced by a simple scene description); removing the memory bank further reduces temporal stability.
A small amount of human feedback measurably improves interaction reconstruction quality (per ablation study).
HAT-4D-generated data improves baseline model performance when used for fine-tuning.
Qwen3-VL interaction rankings show strong correlation with human judgments (user study validation).

Limitations¶

Remains ill-posed for highly ambiguous monocular scenarios even with HITL; human feedback is required at sparse keyframes, introducing a manual cost.
4D propagation operator (L4GM) was trained on synthetic data; domain gap may still limit quality on extreme real-world deformations.
MVOIK-4D is relatively small (77 tasks, 112 scenarios); generalization claims are limited by dataset scale.
Evaluation of interaction quality relies on Qwen3-VL as a judge, which may inherit VLM biases despite user-study correlation.
The table in the provided text is truncated; exact HAT-4D metric scores are not fully visible.

Relevance to Vision-Language Models¶

VLMs are the semantic backbone of HAT-4D: Qwen3-VL performs the entire IKG construction (parsing physical attributes, interaction semantics, depth ordering, and event segmentation from raw video) and serves as the interaction quality judge during evaluation. This exemplifies how VLMs can be promoted from passive perception tools to active causality engines within agentic pipelines, a paradigm shift relevant to anyone tracking VLM applications in embodied/3D settings. The framework also demonstrates HITL prompting strategies for injecting human corrections as system prompts for online adaptation, a lightweight alternative to full fine-tuning that is directly applicable to VLM-centric agent workflows.