Skip to content

HAT-4D: Lifting Monocular Video for 4D Multi-Object Interactions via Human-Agent Collaboration

🕒 Published (v1): 2026-06-26 16:05 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HAT-4D is a human-in-the-loop multi-agent framework that reconstructs physically plausible 4D multi-object interactions (geometry, dynamics, physics) from a single monocular video. It introduces an Interaction Knowledge Graph (IKG) as a structured causal engine to guide VLM-driven 3D generation and 4D propagation agents, and provides the MVOIK-4D benchmark with a multi-dimensional evaluation protocol.

Problem

Monocular 4D reconstruction of multi-object interactions is fundamentally ill-posed due to severe occlusions, depth ambiguities, and complex evolving dynamics. Existing approaches fall into two dead-ends: hardware-intensive multi-camera rigs that are expensive and unscalable, or generative models (SDS-based or end-to-end) that primarily handle isolated/stylized assets and suffer severe domain gaps on real-world interactions (implausible deformations, floating interactions, temporal jittering).

Method

The framework (Fig. 2) operates in several stages orchestrated by VLM-driven agents:

  • Interaction Knowledge Graph (IKG) (Fig. 3): Given an input monocular video, Qwen3-VL digests the sequence into a dynamic directed graph \(G = (O, E, R)\). Object entities \(O\) carry semantic/physical attributes (color, deformability, affordance). Interaction event segments \(E\) partition the video by state transitions (e.g., Contact → Banana Split → Separate), each with a keyframe. Spatio-temporal relations \(R\) encode interacting pairs, semantic predicates \(p\) with confidence \(c\) and distance hint \(d\), depth ordering \(O_{\text{depth}}\), and categorical relative position \(S\).

  • 3D Object Generation & Composition: The agent produces virtual anchors per entity, invokes SAM3D to reconstruct them as 3D Gaussian Splats, consolidates multi-anchor instances, then composes assets into a coherent scene via lightweight 6DoF pose optimization guided by IKG spatial timelines. Contact points are computed for physical plausibility.

  • Memory-Augmented 4D Propagation: Composed 3D keyframes are cached in a memory bank. The segment-wise 4D Propagation Operator (L4GM) renders 4 orthogonal planes from the 3D composition as spatial initialization, conditioned on both the preceding refined frame and keyframes in the memory bank for long-term temporal stability.

  • Multi-Dimensional Evaluation & Rollback: A critic agent performs dynamic assessment (interpenetration, memory stability) and static assessment (visual fidelity, cross-view consistency), triggering selective rollbacks — physics errors replay at 4D propagation, visual errors roll back to 3D object generation.

  • Multi-Level Human-in-the-Loop: Three refinement operators — Gaussian-level (position/orientation/color/opacity of individual splats), region-level (local latent denoising via multi-view video models), and object-level (re-generate via SAM3D + pose adjustment). Corrections are fed back as system prompts for online fine-tuning of agents.

Key Contributions

  • First agentic framework for monocular 4D multi-object interaction reconstruction that jointly recovers geometry, temporal dynamics, and physical interactions.
  • Interaction Knowledge Graph (IKG): a temporally-conditioned directed graph that explicitly encodes long-term physical changes and interaction cues to guide generation.
  • MVOIK-4D benchmark: 77 tasks across 112 interaction scenarios, with a multi-dimensional evaluation protocol (overall quality, temporal memory, interaction reconstruction quality).
  • Multi-level human-in-the-loop collaboration scheme with online fine-tuning of VLM agents and three granularities of refinement operators.

Results

  • SOTA on most evaluation metrics on the MVOIK-4D benchmark (Tab. 1): best LPIPS and FVD for overall generation quality, best Deform and Relation for interaction quality, best Intra for frame-to-frame memory quality.
  • Ablations: removing IKG or memory bank degrades performance across all dimensions.
  • A small amount of human feedback improves interaction reconstruction.
  • Data produced by HAT-4D effectively improves baseline performance when used for fine-tuning.
  • Strong correlation between Qwen3-VL-based evaluation and human judgments (user study in appendix).

Limitations

  • Relies on specific VLM (Qwen3-VL) and generation operators (SAM3D, L4GM) — performance bounds are coupled to these base models.
  • Human-in-the-loop requires manual correction at sparse keyframes; while minimal, it still creates a bottleneck for fully automated large-scale data production.
  • The benchmark (MVOIK-4D), while real-world, is captured with a multi-camera platform (calibrated views) — the distribution may not fully cover truly in-the-wild monocular video conditions.
  • No fully automated variant without any human feedback is benchmarked as the primary system (ablated only).
  • Scalability to scenes with very large numbers of interacting objects or long video durations is not empirically characterized.

Relevance to Agentic AI / LLM Agents

HAT-4D exemplifies a structured multi-agent design for a hard vision+physics task: it decomposes 4D reconstruction into subtasks (parsing, generation, composition, propagation, evaluation) each handled by specialized agents coordinated through a shared symbolic graph (IKG). The human-in-the-loop refinement shows how VLM agents can handle most of the pipeline autonomously while humans intervene only at sparse keyframes, suggesting a general pattern for scaling data production in embodied AI. The IKG as a causal "constraint engine" between agents is a concrete architectural contribution for agentic workflows.