Skip to content

HAT-4D: Lifting Monocular Video for 4D Multi-Object Interactions via Human-Agent Collaboration

🕒 Published (v1): 2026-06-26 16:05 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HAT-4D is an agentic framework that reconstructs physically plausible, temporally consistent 4D multi-object interactions from a single monocular video by coupling VLM-driven specialized agents with a multi-level human-in-the-loop (HITL) feedback mechanism. It introduces the Interaction Knowledge Graph (IKG) as a structured causality engine that encodes object entities, temporal event segments, and spatio-temporal relations to guide 3D generation and 4D propagation. The framework also serves as a scalable data engine for constructing MVOIK-4D, the first benchmark for open-world monocular 4D interaction reconstruction.

Problem

Existing monocular 4D reconstruction methods target isolated objects and break down under the severe mutual occlusions and complex deformation dynamics of multi-object interactions. Multi-camera capture rigs are prohibitively expensive and unscalable to in-the-wild settings, while generative approaches trained on synthetic data suffer large domain gaps (floating interactions, temporal jitter, implausible deformations) when applied to real-world object interactions. No benchmark exists for evaluating physical plausibility and temporal memory in this setting.

Method

Given an input monocular video, a Qwen3-VL agent parses the sequence into an Interaction Knowledge Graph \(G = (O, E, R)\), where \(O\) is the set of object entities with physical attributes, \(E = \{e_1, \ldots, e_m\}\) partitions the video into temporally bounded interaction event segments (keyed on state transitions such as topological changes or occlusion shifts), and \(R\) encodes per-segment spatio-temporal relations as tuples \(r_i = \langle (o_a, o_b), I, O_\text{depth}, S \rangle\) covering interaction semantics, depth ordering, and relative 3D position.

Downstream, HAT-4D orchestrates three specialized agent skills guided by the IKG: 1. 3D Object Generation Skill — SAM3D localizes and reconstructs each entity as a 3D Gaussian Splat at the first frame and at IKG-flagged keyframes \(\ddot{T}_i\). 2. 3D Object Composition Skill — lightweight pose optimization aligns entities into a coherent scene with physically plausible contact points; results are cached in a memory bank. 3. Memory-Augmented 4D Propagation — L4GM propagates each event segment conditioned on the memory bank (size 8 frames), rendering 4 orthogonal planes from the composed 3D result as spatial initialization.

A 4D Generation Evaluation Skill acts as critic: it renders multi-view videos of each segment, performs dynamic assessment (interpenetration, long-term memory) and static assessment (texture, cross-view fidelity) against IKG constraints, and triggers selective rollback — localized re-propagation for physics errors or full rollback to the 3D generation stage for visual artifacts.

HITL collaboration operates at three granularities: Gaussian-level (position, color, opacity edits), region-level (local re-generation via latent denoising), and object-level (full re-generation with SAM3D). Human corrections are injected as system prompts enabling online fine-tuning; human-refined assets also serve as pseudo ground-truth for offline fine-tuning of the learnable 4D operator.

Key Contributions

  • HAT-4D framework: first HITL multi-agent system for monocular video → physically plausible 4D multi-object reconstruction.
  • Interaction Knowledge Graph (IKG): a dynamic directed graph \(G=(O,E,R)\) that formalizes long-term physical causality and drives all downstream agent decisions.
  • Multi-level HITL collaboration: three-granularity refinement operators (Gaussian, region, object) plus online fine-tuning from human corrections and offline fine-tuning loop.
  • MVOIK-4D benchmark: 77 tasks across 112 interaction scenarios with two subsets (ToolOIK-4D for deformable interactions; MemoryOIK-4D for occlusion-dependent temporal memory), plus a multi-dimensional protocol covering CLIP/LPIPS/FVD, intra-frame continuity, long-term DINOv3-based memory stability, and Qwen3-VL-judged interaction plausibility.

Results

  • HAT-4D achieves best LPIPS and FVD among all baselines on MVOIK-4D, indicating superior perceptual quality and temporal realism.
  • Baselines for context: L4GM scores CLIP 0.8259 / LPIPS 0.1968 / FVD 962.86 / Deform 2.53 / Relation 1.90 / Intra 0.0013.
  • HAT-4D maintains competitive CLIP (semantic alignment), indicating it does not sacrifice semantics for geometric fidelity.
  • Ablation: removing the IKG (HAT-4D w/o IKG) degrades performance; removing the memory bank (HAT-4D w/o memory) also degrades, confirming both components' contribution.
  • Ablation: introducing even a small amount of human feedback measurably improves interaction reconstruction quality.
  • Human study confirms strong consistency between Qwen3-VL plausibility rankings and human judgments.
  • Fine-tuning baselines on HAT-4D-generated data improves baseline performance, validating its data engine utility.

Limitations

  • Still requires human involvement at sparse keyframes; the system is not fully automatic and human effort scales with scene complexity and VLM failure rate.
  • IKG extraction depends entirely on Qwen3-VL; errors in graph construction (wrong object entities, incorrect depth ordering, misidentified interaction predicates) propagate through the entire pipeline.
  • The 4D propagation operator (L4GM) was trained on synthetic data; real-world domain gap is mitigated but not eliminated.
  • MVOIK-4D covers 112 scenarios, which is small-scale; generalization to long-tail interactions and completely unseen object categories is unverified.
  • Gaussian-level and region-level HITL refinement require users with spatial editing fluency; the quality of human corrections varies.

Relevance to Harnesses / Meta-Harnesses

HAT-4D is a canonical multi-agent harness: it decomposes a complex reconstruction task into discrete skill-agents (3D generation, composition, 4D propagation, evaluation/rollback), orchestrated by a structured knowledge graph that serves as the shared state and constraint engine analogous to a meta-harness's context or plan object. The critic-agent + selective rollback loop is a concrete implementation of the evaluate-repair pattern common in agentic harnesses, and the three-granularity HITL mechanism shows how human-in-the-loop signals can be injected at multiple pipeline stages without restarting the full workflow. The dual-loop learning (online fine-tuning from human corrections; offline fine-tuning from validated outputs) directly instantiates the self-improving data-engine pattern that meta-harnesses increasingly employ to reduce human overhead over time.