Skip to content

EA3D: Online Open-World 3D Object Extraction from Streaming Videos

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EA3D is a unified online framework that simultaneously reconstructs 3D geometry and extracts open-world semantic knowledge from monocular streaming video, requiring no pre-built geometry, camera poses, or annotated training data. It embeds VLM-derived and VFM-derived features into 3D Gaussian primitives via a feed-forward online update strategy, enabling a broad range of downstream 3D tasks in real time.

Problem

Existing 3D scene understanding methods are either (a) offline, requiring complete multi-view captures before reconstruction begins, or (b) dependent on pre-constructed geometry (point clouds, depth maps) and ground-truth or COLMAP-estimated camera poses. VLMs provide strong open-world 2D understanding but suffer from multi-view inconsistencies, geometric misalignment, and occlusion blindness when naively lifted to 3D. No prior method jointly reconstructs and semantically understands 3D scenes in a purely online, pose-free, open-vocabulary setting.

Method

EA3D has three tightly coupled components:

Knowledge-Integrated Feature Map. At each timestep, a VLM (e.g., GPT-4o) is prompted to enumerate all objects and their physical properties; new categories are appended to a dynamic semantic cache Ω. A CLIP text encoder embeds categories into a continuous vector; a CLIP visual encoder and GroundedSAM encoder produce pixel-level feature maps. Binary masks from category-feature similarity are used to aggregate and normalize per-instance features into a unified knowledge-integrated feature map F_t. Cross-frame matching distributions (softmax over dot-product similarity between adjacent F_t) propagate Gaussian features via single-step forward warping.

Online Feature Gaussians. Camera poses are estimated incrementally using Cut3R (dense point maps) fused with sparse keypoints, maintained in an online keypoint graph with local bundle adjustment to reduce drift. Per-frame inverse depth and point maps initialize new Gaussian primitives; co-visible regions share translation/rotation parameters to enforce inter-frame consistency; low-contribution high-opacity Gaussians are pruned; adaptive one-step splitting handles under-reconstructed regions.

Recurrent Joint Optimization. A semantic-awareness regularization term L_δ penalizes scale variance among Gaussians of the same semantic category, reducing redundancy. The full training loss combines L1 photometric loss, rendered-vs-estimated depth L1, L2 knowledge-feature consistency loss, and L_δ. All Gaussian parameters (geometry + features) and poses are jointly optimized over the causal window of frames seen so far.

A multi-level decoder renders RGB, segmentation, 3D bounding boxes, semantic occupancy, and mesh from the accumulated Gaussian features.

Key Contributions

  • First unified online framework for simultaneous 3D reconstruction and open-world scene understanding without geometry or pose priors.
  • Dynamic semantic cache with feed-forward VLM+VFM feature embedding into 3D Gaussian primitives, enabling causal knowledge transfer across streaming frames.
  • Online visual odometry (Cut3R + sparse SLAM) integrated with incremental Gaussian addition/pruning for pose-free monocular operation.
  • Semantic-aware regularization that links Gaussian geometry to semantic identity, improving 3D consistency and reducing redundant primitives.
  • Multi-task outputs (photo-realistic rendering, instance/semantic segmentation, 3D bbox, occupancy, mesh) from a single unified representation.

Results

  • ScanNet multi-task (Table 1): EA3D achieves PSNR 25.8 / SSIM 0.89 / mIoU 46.3 / mAcc 71.8 / AP 57.9 / mAP 39.9 / IoU 55.4 / Occ-mIoU 33.9, outperforming best offline baseline FeatureGS (PSNR 23.9, mIoU 41.1) and best enhanced baseline FeatureGS+HiCOM (PSNR 24.5, mIoU 40.8).
  • LERF sparse/incremental (Table 2): At 70 views, EA3D mIoU 57.4 vs. FeatureGS 53.6 and LangSplat 51.5, while operating online at 0.235 FPS vs. offline baselines at 0.005–0.018 FPS.
  • Efficiency (Fig. 4b): EA3D achieves higher mIoU than all methods while reducing per-frame optimization time; rendering speed ~210 FPS (Table 3 ablation).
  • Ablation (Table 3): Removing CLIP encoder drops mIoU 46.3→41.6; removing SAM encoder 46.3→42.8; removing semantic-aware regularization 46.3→44.3; removing online Gaussian update 46.3→44.5; removing online odometry 46.3→45.4; removing joint optimization 46.3→45.7.

Limitations

  • Per-frame training throughput (0.235 FPS) is far below real-time; unsuitable for latency-critical deployment.
  • Relies on Cut3R for pose estimation, whose accumulated drift requires iterative correction that adds computational overhead.
  • VLM hallucinations or missed detections propagate into the semantic cache and downstream 3D representations with no explicit error-recovery mechanism.
  • Evaluated only on static indoor scenes (ScanNet, LERF); dynamic objects (moving people, articulated bodies) are not addressed.
  • Requires a single A100 80 GB GPU; memory footprint scales with accumulated Gaussians.

Relevance to Vision-Language Models

EA3D addresses the well-documented failure of VLMs in 3D settings—multi-view inconsistency, geometric misalignment, and occlusion blindness—by using VLM outputs (object labels, physical properties) as semantic priors that are grounded into spatially consistent 3D Gaussian representations through online reconstruction. This is a concrete instantiation of how VLMs can be coupled with differentiable 3D representations to go beyond 2D image-level understanding. For researchers tracking VLMs, it demonstrates a practical pipeline for lifting open-vocabulary VLM knowledge into continuous 3D scene representations, with direct relevance to embodied AI, robot manipulation, and AR/VR where VLMs must reason about 3D space from monocular streams.