Skip to content

Robust Cross-modal Alignment Learning for Cross-Scene Spatial Reasoning and Grounding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces Cross-Scene Spatial Reasoning and Grounding (CSSRG), a new task requiring an agent to locate a described 3D object anywhere across a large archive of scenes rather than a single known scene. The authors propose CoRe, a matching-then-grounding framework with two modules addressing the dual challenges of prohibitive traversal cost and partial text-scene alignment. They also release CrossScene-RETR, a richer benchmark dataset built atop ScanNet with GPT-4o-generated discriminative descriptions.

Problem

Existing 3D Visual Grounding (3DVG) methods assume the target object's scene is pre-determined, making them inapplicable when an agent must search hundreds of previously visited environments. Naively extending 3DVG to this open-scene setting causes ~75× inference slowdown and performance collapse to 11.7% of single-scene accuracy (on VisTA/ScanRefer with 141 rooms). Additionally, object descriptions exhibit partial alignment with scenes—they describe only the target and its immediate context, not the full scene—causing false-negative pairs during training.

Method

CoRe adopts a two-stage matching-then-grounding pipeline:

  1. Robust Text-Scene Aligning (RTSA): Fine-grained word and object features are aggregated into global common-space representations via complementary feature aggregation (two views emphasizing discriminative tokens vs. informative dimensions). A complementary learning loss (GCE-expanded) trains only on negative pairs, avoiding noisy partially-aligned positives. The robustness parameter q is set adaptively to the pair's similarity, downweighting likely false negatives.

  2. Tailored Word-Object Associating (TWOA): Within the retrieved scene only, word and object features are jointly processed through a Screening Attention (ScA) mechanism. ScA sorts attention scores into L quantile segments and progressively accumulates them coarse-to-fine with learnable coefficients {µ_j}, pruning low-attention (irrelevant) word-object associations while retaining discriminative ones—yielding non-redundant fine-grained alignment.

A targeted masking strategy during encoding masks attributive/spatial vocabulary tokens in text and class-semantic/positional embeddings in point clouds to improve feature robustness. The overall loss is L = L_c + λ_m L_m + λ_g L_g.

Key Contributions

  • Formalization of the CSSRG task: grounding described objects across an unconstrained archive of 3D scenes without pre-known text-scene correspondence.
  • CoRe framework with RTSA (robust cross-modal scene retrieval via complementary learning on negatives only) and TWOA (progressive Screening Attention for word-object grounding).
  • CrossScene-RETR dataset: GPT-4o-generated descriptions averaging 77.7 words, 23.2 information points, and 11.4 referenced objects per query—substantially richer than ScanRefer (4.5 info points, 1.8 objects) and annotated with conspicuous/regular/confusing difficulty tiers.

Results

  • On CrossScene-RETR (Overall Acc@0.25 / Acc@0.5): CoRe 22.99 / 21.26 vs. ESA+GPS 15.60 / 14.44 vs. HREM+VisTA 17.25 / 15.96; inference at 54 ms vs. 1814 ms (HREM+VisTA) and 27.2 s (GPS alone).
  • On ScanRefer scene-matching R@1: CoRe 13.29 vs. best CMM baseline CRCL-C 10.78; object grounding Acc@0.25: CoRe 5.86 vs. CRCL-F 5.71.
  • On Nr3D R@1: 14.56 vs. CRCL-F 13.95; object grounding: CoRe 5.86 vs. CRCL-F 5.52.
  • CoRe achieves ~250Ă— speedup over 3DVG baselines (12–27 s per query) for cross-scene inference.
  • Ablation on CrossScene-RETR: removing RTSA drops R@Sum from 186.45 to 5.25; replacing ScA with dense Self-Attention drops Acc@0.25 from 22.99 to 19.76; replacing ScA with Cross-Attention drops to 17.53.

Limitations

  • Performance on CSSRG remains low overall (best Acc@0.5 ~21%), indicating the task is far from solved.
  • CrossScene-RETR descriptions are GPT-4o-generated; manual verification cannot fully eliminate all hallucinated or unnatural descriptions.
  • The two-stage pipeline may propagate scene retrieval errors to grounding—if the correct scene is not retrieved, grounding necessarily fails.
  • Scene archive is sourced solely from ScanNet indoor scenes; generalization to outdoor or larger-scale environments is untested.
  • Effectiveness of adaptive q (proxy similarity as reliability) is heuristic and not theoretically grounded.

Relevance to Agentic AI / LLM Agents

CSSRG directly targets the practical scenario where a long-deployed embodied agent must localize objects across its full memory of visited environments given open-vocabulary natural language queries—a prerequisite for realistic task planning and navigation in smart homes and robotics. The matching-then-grounding paradigm is an important efficiency pattern for agent memory retrieval: reducing exhaustive scene traversal to a fast retrieval stage mirrors how language agents use vector retrieval before reasoning. The partial-alignment robustness problem addressed by RTSA mirrors challenges in RAG systems where query-document correspondence is inherently noisy. CrossScene-RETR provides a concrete benchmark for evaluating cross-scene spatial reasoning, a capability gap in current 3D vision-language agents.