Skip to content

3D-Agent: A Tri-Modal Multi-Agent Responsive Framework for Comprehensive 3D Object Annotation

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Tri-MARF is a three-agent pipeline for automated 3D object annotation that fuses multi-view RGB images, text, and point clouds. It uses a VLM agent (Qwen2.5-VL-72B) for per-view caption generation, a multi-armed bandit agent for cross-view description aggregation, and a cosine-similarity gating agent for point-cloud/text alignment. On Objaverse-LVIS it achieves a CLIPScore of 88.7 vs. 78.6–82.4 for SOTA baselines at 12,000 objects/hour on a single A100.

Problem

Single-model 3D annotation methods fail to simultaneously handle viewpoint inconsistency, occlusion, geometric complexity, and semantic hallucination. Existing VLM-based approaches (Cap3D, 3D-LLM, GPT4Point) produce incomplete or inconsistent captions because no single model can optimize accuracy, completeness, consistency, and throughput at once.

Method

Tri-MARF decomposes annotation into four sequential stages:

  1. Data Preparation — six standard views (front/back/left/right/top/bottom) are rendered; point cloud features are extracted.
  2. VLM Annotation (Agent 1) — Qwen2.5-VL-72B-Instruct generates M=5 candidate captions per view via multi-turn structured prompting (viewpoint-aware ID → attribute elicitation → contextual integration) at temperature=0.7; token log-probabilities produce a confidence score per caption.
  3. Information Aggregation (Agent 2) — BERT embeddings + DBSCAN cluster semantically redundant candidates; CLIP visual-text alignment scores each cluster representative; a multi-armed bandit (UCB1) adaptively selects the best description per view over time by treating each candidate as a bandit arm and maximizing a composite reward of VLM confidence and CLIP similarity. Front/back views receive priority weighting in the global description fusion.
  4. Gating (Agent 3) — A pre-trained point cloud encoder (Uni3D) and CLIP text encoder produce embeddings; cosine similarity against threshold α=0.557 either accepts the annotation or flags it for manual review.

Key Contributions

  • Tri-modal multi-agent architecture combining 2D multi-view images, text, and 3D point clouds for annotation.
  • Multi-armed bandit (UCB1) formulation for adaptive cross-view description selection, replacing static voting or rule-based aggregation.
  • Token log-probability confidence metric for uncertainty-aware candidate ranking.
  • Point-cloud gating stage that catches VLM hallucinations via geometric-semantic consistency checking.
  • Throughput of 12,000 objects/hour on a single A100; has annotated ~2 million 3D models.

Results

  • CLIPScore: 88.7 (Tri-MARF) vs. 82.4 (human), 80.1 (ScoreAgg), 78.6 (Cap3D) on Objaverse-LVIS 1k; 86.1 vs. 81.0/78.5/76.4 on Objaverse-XL 5k; 82.3 vs. 78.9/76.2/74.8 on ABO 6.4k.
  • ViLT R@5 (I2T/T2I): 45.2/43.8 vs. 40.0/38.5 (human), 37.8/36.0 (ScoreAgg), 35.2/33.4 (Cap3D) on Objaverse-LVIS.
  • Type annotation (GPT-4o semantic accuracy): 98.32% vs. 95.72% (human), 96.78% (ScoreAgg), 92.72% (Cap3D) on Objaverse-LVIS.
  • Generalization (zero-shot on ShapeNet-Core/ScanNet/ModelNet40): CLIPScore drops only 7.2% vs. 9.8–15% for competitors.
  • Throughput: 12k objects/hour vs. 9k (ScoreAgg), 8k (Cap3D), 6.5k (3D-LLM).
  • 6 views optimal; performance degrades at 8 views due to redundancy.

Limitations

  • Depends on Qwen2.5-VL-72B-Instruct; smaller VLMs are tested in supplementary but the primary system is compute-heavy.
  • Gating threshold (α=0.557) is empirically set via grid search on validation data; may not generalize to all 3D domains.
  • MAB convergence requires sufficient trials; cold-start behavior on novel object categories is not characterized.
  • Point cloud inputs assumed clean/pre-rendered; robustness to real-world noisy scans is only partially evaluated.
  • Multi-agent communication strategies are acknowledged as underdeveloped; future work is planned.

Relevance to Agentic AI / LLM Agents

Tri-MARF is a concrete instantiation of the specialist multi-agent paradigm: complex tasks are decomposed into sub-tasks matched to agent strengths (vision-language understanding, bandit-based selection, geometric verification) rather than attempting a monolithic solution. The use of reinforcement learning (UCB1 bandit) to coordinate agent decision-making at the information aggregation stage—learning dynamically rather than using fixed rules—is an important design pattern for robust agentic pipelines. The gating agent as an explicit hallucination-suppression mechanism illustrates how agent pipelines can incorporate verifier roles. For researchers tracking LLM agent architectures, this paper demonstrates that lightweight RL-based coordination (not full LLM-to-LLM communication) can drive measurable gains in a production-scale system (~2M annotated objects).