Skip to content

3D-Agent: A Tri-Modal Multi-Agent Responsive Framework for Comprehensive 3D Object Annotation

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Tri-MARF is a tri-modal multi-agent framework for automated 3D object annotation that combines VLM-generated multi-view descriptions, reinforcement learning-based description selection (multi-armed bandit), and point cloud-text gating. It achieves state-of-the-art CLIPScore of 88.7 on Objaverse-LVIS while annotating 12,000 objects/hour on a single A100, surpassing both prior automated methods and human annotations in semantic precision.

Problem

Single-model 3D annotation approaches struggle with viewpoint inconsistency, occlusion, and hallucination because no single model can simultaneously optimize semantic accuracy, cross-view consistency, geometric fidelity, and throughput. Existing VLM-based methods (Cap3D, ScoreAgg, 3D-LLM) apply 2D image understanding without grounding in 3D geometry, leading to hallucinated or inconsistent captions.

Method

Tri-MARF decomposes annotation into four sequential stages handled by three specialized agents:

  1. VLM Annotation Agent: Renders six standard views (front/back/left/right/top/bottom) of each 3D object and queries Qwen2.5-VL-72B-Instruct with a multi-turn prompting protocol (viewpoint-aware ID → attribute elicitation → contextual integration). Generates M=5 candidate descriptions per view using temperature=0.7 sampling; retains token-level log-probabilities as a confidence score (mean token log-likelihood).

  2. Information Aggregation Agent: Clusters the 5 candidates per view using BERT embeddings + DBSCAN cosine similarity to remove paraphrases, then re-scores each cluster representative via CLIP image-text alignment (softmax-normalized cosine similarity). A multi-armed bandit (UCB1) selects the best cross-view description over time, learning to balance semantic distinctiveness and visual grounding. Front/back views receive priority weighting in the final global description assembly.

  3. Gating Agent: Encodes the aggregated text caption and the object's 3D point cloud (via pretrained encoders) into a shared space; computes cosine similarity and applies threshold α=0.557. Captions below threshold are flagged for manual review, suppressing VLM hallucinations that pass the 2D stage.

Key Contributions

  • Tri-modal input integration (2D multi-view images + text + 3D point clouds) within a multi-agent pipeline for 3D object captioning.
  • Token log-probability confidence metric for principled uncertainty quantification in VLM outputs.
  • UCB1 multi-armed bandit for adaptive cross-view description selection, replacing static voting or top-k heuristics.
  • Point cloud–text cosine similarity gating agent that grounds captions in 3D geometry and filters hallucinations.
  • Large-scale deployment: ~2 million 3D models annotated to date at 12k objects/hour.

Results

  • CLIPScore: 88.7 (Tri-MARF) vs. 82.4 (human), 80.1 (ScoreAgg), 78.6 (Cap3D) on Objaverse-LVIS 1k.
  • ViLT R@5 (I2T/T2I): 45.2/43.8 vs. 40.0/38.5 (human), 37.8/36.0 (ScoreAgg), 35.2/33.4 (Cap3D) on Objaverse-LVIS.
  • Type inference (GPT-4o semantic score): 98.32% vs. 95.72% (human), 96.78% (ScoreAgg), 92.72% (Cap3D) on Objaverse-LVIS.
  • Throughput: 12,000 objects/hour (Tri-MARF) vs. 9k (ScoreAgg), 8k (Cap3D), 6.5k (3D-LLM).
  • Cross-dataset generalization (ShapeNet-Core, ScanNet, ModelNet40): Tri-MARF ranks second only to human annotation; CLIPScore drops only 7.2% vs. 11.5% (Cap3D) and 9.8% (ScoreAgg) from in-distribution to out-of-distribution.
  • Optimal viewpoints: 6 views peak for all methods; adding an 8th view degrades performance due to redundancy.
  • ABO (6.4k): CLIPScore 82.3 vs. 78.9 (human), 76.2 (ScoreAgg), 74.8 (Cap3D).

Limitations

  • The gating threshold α=0.557 is empirically tuned on validation data; performance on domains with very different point cloud distributions is not guaranteed.
  • Manual review is still required for samples below the gating threshold; the paper does not quantify what fraction of samples fall below threshold.
  • Relies on a 72B-parameter VLM (Qwen2.5-VL-72B-Instruct), which limits accessibility for resource-constrained settings despite the fast per-object throughput on A100.
  • The MAB reward signal depends on VLM confidence scores and CLIP similarity, inheriting biases from those models.
  • Evaluation is limited to object-level annotation; scene-level or part-level generalization is not demonstrated in the main paper.

Relevance to Vision-Language Models

This paper directly addresses a core failure mode of VLMs—hallucination and cross-view inconsistency—by embedding them as one agent in a larger system rather than relying on them end-to-end, which is a practically important design pattern for VLM deployment. The point cloud gating mechanism demonstrates how grounding VLM outputs in a non-visual modality (3D geometry) can filter hallucinations that are invisible to image-text metrics alone. For researchers tracking VLMs, the multi-armed bandit strategy for aggregating multiple stochastic VLM samples offers a reusable alternative to simple majority voting or top-1 selection in annotation pipelines. The benchmark results—where Tri-MARF exceeds human CLIPScore on ABO—also raise relevant questions about what CLIPScore actually captures versus genuine semantic quality.