IGGT4D: Streaming 4D Instance-Grounded Geometry Transformer¶
🕒 Published (v1): 2026-07-21 16:00 UTC · Source: Arxiv · link
Why this paper was selected
Streaming 4D instance-grounded geometry; spatial intelligence backbone for embodied agents
Ask a follow-up
Open an assistant pre-loaded with this paper's context.
TL;DR¶
IGGT4D is a causal streaming Transformer that jointly predicts camera pose, dense 3D geometry, and temporally consistent instance identity from sequential video frames without any per-scene optimization. It introduces an efficient two-stage streaming clustering algorithm and the InsScene4D-147K dataset (147K sequences, 60M masks) to supervise joint geometry-instance learning. The system supports instance spatial tracking, open-vocabulary segmentation, and scene grounding as downstream applications.
Problem¶
Existing streaming spatial foundation models (CUT3R, Stream3R, Spann3R) are geometry-centric: they predict depth and camera pose but do not maintain persistent object identities across frames. Conversely, 3D-aware vision-language methods that do handle semantics rely on externally extracted 2D features and per-scene optimization, preventing unified geometry-instance learning in long dynamic sequences. A compounding gap is the absence of large-scale 4D supervision providing metric geometry, camera motion, and temporally consistent instance masks together.
Method¶
IGGT4D adapts the DA3 unified geometric Transformer to causal streaming inference by applying causal masks to all cross-camera and cross-view attention, so each frame attends only to current and past observations. KV caches store historical geometry and cross-view context for reuse without redundant recomputation.
For each frame \(t\), the model outputs camera parameters \(\pi_t \in \mathbb{R}^9\), a ray map \(R_t \in \mathbb{R}^{H_r \times W_r \times 6}\), a depth map \(D_t \in \mathbb{R}^{H \times W}\), and an instance feature map \(S_t \in \mathbb{R}^{H \times W \times 8}\) via a Tri-DPT head that jointly decodes all three modalities. The instance branch receives geometry-aware attention from the depth and ray branches to ground instance embeddings in 3D structure.
Online instance association uses two-stage streaming clustering: (1) intra-frame clustering groups pixels by cosine similarity above threshold \(\tau_s\) into a core region, expanded via connected components at a looser threshold \(\tau_l\); (2) local cluster centers are matched to a global instance codebook \(C_t = \{(c_k, a_k)\}\) maintained across frames. Matched entries update centers via area-weighted fusion: $\(c_k \leftarrow \mathrm{norm}\!\left(\frac{a_k c_k + |M_{t,k}| c_{t,i}}{a_k + |M_{t,k}|}\right), \quad a_k \leftarrow a_k + |M_{t,k}|\)$ This replaces the quadratic offline HDBSCAN used by the predecessor IGGT.
Training uses a geometry loss \(\mathcal{L}_\mathrm{geo} = \mathcal{L}_D + \mathcal{L}_R + \mathcal{L}_P + \mathcal{L}_\pi\) (depth L1 + confidence regularization + gradient consistency + ray L1 + 3D point L1 + camera L1) and a multi-view contrastive instance loss \(\mathcal{L}_\mathrm{ins}\) with intra-view pull/push margins and cross-view matching constraints across frame pairs.
InsScene4D-147K is built via a three-step pipeline: (1) TSDF-fused 3D mesh reconstruction from DA3 depth predictions, (2) per-frame mesh vertex projection with depth-consistency visibility filtering and global ID inheritance maps, (3) SAM2 category-agnostic segmentation matched to inherited IDs by IoU, with new instances initialized for unmatched regions.
Key Contributions¶
- Causal streaming 4D formulation: recasts geometry-instance reconstruction as online sequential prediction with KV-cache reuse, enabling constant-memory long-sequence inference.
- IGGT4D model: a unified streaming Transformer combining a causal geometry backbone, a Tri-DPT multi-branch decoding head, and an online instance codebook with area-weighted center updates.
- InsScene4D-147K dataset: 147K video sequences across static-real, static-synthetic, dynamic-real, and dynamic-synthetic domains with RGB, depth, poses, point clouds, and 4D-consistent instance masks (60M masks total), generated by an automated geometry-guided annotation pipeline.
- Downstream 4D applications: instance spatial tracking, open-vocabulary segmentation via aggregation of 2D VLM features over 4D-consistent masks, and LMM-driven scene grounding.
Results¶
Camera Pose Estimation (AUC@3 / AUC@30, averaged over HiRoom, ETH3D, 7Scenes, ScanNet++, Table 1a): - IGGT4D (streaming): 0.4464 / 0.8924 — best among all streaming baselines; surpasses LingBot-Map (0.3060 / 0.8647), Stream3R (0.1871 / 0.7947), CUT3R (0.1022 / 0.6977) - Narrows gap to best offline model DA3 (0.6046 / 0.9352)
3D Reconstruction F1 (without / with ground-truth poses, Table 1b): - IGGT4D: 0.6678 / 0.7225 — best among streaming models; next best LingBot-Map (0.5228 / 0.6082)
Instance Spatial Tracking (T-mIoU / T-SR, long sequences ~100 frames, Table 2a): - IGGT4D: 58.84 / 85.41 — only method that avoids OOM; IGGT runs out of memory on all long-sequence splits; SAM2: 45.38 / 58.65 - Short sequences (~16 frames): IGGT4D 59.25 / 86.28 vs IGGT 51.68 / 84.65
Open-Vocabulary Segmentation (mIoU / mAcc, Table 3, long sequences): - IGGT4D achieves highest mIoU and mAcc among all baselines including OpenSeg, LSeg, Feature-3DGS, and IGGT on both Waymo and ScanNet++ (exact numbers in table are partially truncated in provided text)
Limitations¶
- Causal streaming still lags offline full-attention models (DA3) on pose estimation (AUC@3: 0.4464 vs 0.6046) and reconstruction, due to the absence of bidirectional cross-frame reasoning.
- Dynamic scene annotation in InsScene4D-147K relies on available ground-truth dynamic-object masks (HOI4D), limiting scalability to scenes without such annotations.
- The 8-dimensional instance feature space and codebook-based clustering may struggle with very large scene graphs containing many simultaneously visible objects.
- No explicit motion modeling: object reidentification relies entirely on feature similarity, potentially breaking under large appearance changes or rapid motion.
- The predecessor IGGT (offline) runs OOM on long sequences, but performance comparison on short sequences shows IGGT4D outperforming; no analysis of failure cases at the transition from short to long sequences.
Relevance to Vision-Language Models¶
IGGT4D directly interfaces with 2D VLMs (e.g., OpenSeg, LSeg) by aggregating their per-frame language features \(f^\mathrm{lang}_{t,k}\) over 4D-consistent instance masks, converting view-dependent VLM predictions into spatially and temporally grounded language embeddings — addressing a core limitation of VLMs that lack metric 3D grounding. The system also provides geometry-anchored instance tokens to Large Multimodal Models for 4D scene grounding tasks (dynamic object segmentation, out-of-view reasoning), showing a concrete pathway for connecting streaming video understanding to LMMs. For researchers tracking VLMs, this work demonstrates that tight coupling with geometry-first representations substantially improves open-vocabulary segmentation over standalone 2D VLM baselines, motivating architectures that learn geometry and language jointly rather than using 2D VLMs as detached upstream modules. The InsScene4D-147K dataset, with its temporally consistent instance masks, could also serve as supervision for training spatially grounded VLMs on dynamic scenes.