Do 3D Large Language Models Really Understand 3D Spatial Relationships?¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; exposes 3D LLMs do not truly understand spatial relationships

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Current 3D Large Language Models (3D-LLMs) do not genuinely understand 3D spatial relationships: a blind language model fine-tuned on text-only QA pairs matches or surpasses state-of-the-art 3D-LLMs on SQA3D, exposing critical benchmark shortcut bias. The authors introduce Real-3DQA, a debiased benchmark that filters out linguistically solvable questions and adds viewpoint-rotation consistency testing, and propose a 3D-Reweighted Fine-Tuning (3DR-FT) strategy to force models to rely on 3D visual cues.

Problem¶

Existing 3D situated QA benchmarks (SQA3D, ScanQA, MSR3D) are contaminated with linguistic and commonsense priors: models achieve competitive accuracy without any 3D input. Prior debiasing attempts (e.g., answer-distribution balancing in SQA3D) fail to remove deeper annotation artifacts such as saliency preferences, canonical layouts, and easily guessable answers. No existing benchmark tests cross-viewpoint spatial consistency, making it impossible to distinguish genuine 3D understanding from shortcut learning.

Method¶

Real-3DQA Benchmark Construction: 1. 3D-independence filtering: Three 3D-LLMs ($M_A, M_B, M_C$) are each paired with a blind counterpart $M_X^{\text{blind}}$ fine-tuned without 3D input. Questions answered correctly by both the full and blind model for any of the three systems are labeled 3D-independent and removed ($Q_{\text{3D-filtered}} = Q_A \cup Q_B \cup Q_C$). Questions also answerable by GPT-4o-mini from text alone are further removed, yielding $Q_{\text{final}} = Q' \setminus Q_{\text{GPT}}$. 2. Viewpoint Rotation Score (VRS): Each surviving question is augmented at 90°, 180°, 270° rotations via GPT-4o using scene graphs; directional terms in the situation and answer are updated to match the new reference frame. VRS is defined as $\text{VRS} = \frac{1}{4}\sum_{k=1}^{4} P_k$, where $P_k = \frac{N_k}{N_{\text{total}}} \times 100$ is the percentage of instances with at least $k$ of 4 rotation variants answered correctly.

3D-Reweighted Fine-Tuning (3DR-FT): Per-token loss weights are computed as the ratio of the blind model's surprise to the current model's surprise: $$w_j(y, x_{\text{text}}) = \frac{\log p_\phi(y_j | y_{<j}, x_{\text{text}})}{\log p_\theta(y_j | y_{<j}, x_{\text{text}})}$$ Tokens harder for the blind model receive higher weights in the training loss: $$\mathcal{L}_{\text{3DR-FT}}(\theta) = \mathbb{E}_D\!\left[-\sum_{j=1}^T w_j(y, x_{\text{text}}) \log p_\theta(y_j | y_{<j}, x_{\text{text}}, x_{\text{3D}})\right]$$ This adaptively downweights linguistically trivial samples and upweights spatially demanding ones.

Key Contributions¶

Real-3DQA benchmark: Derived from SQA3D with model-based debiasing that removes questions solvable by text alone and by GPT-4o-mini; includes 11 question categories.
Viewpoint Rotation Score (VRS): A cross-question consistency metric testing whether models maintain correct answers across 0°/90°/180°/270° egocentric perspective shifts.
Diagnostic finding: Blind text-only fine-tuning matches or surpasses 3D-LLMs on SQA3D, ScanQA, and MSR3D, demonstrating that reported benchmark progress does not reflect 3D spatial reasoning.
3DR-FT training strategy: Per-token reweighting based on blind-vs-full model surprise ratio to increase 3D context dependency during fine-tuning.

Results¶

On Real-3DQA, all models suffer dramatic performance drops vs. SQA3D: 3D-LLM drops from 47.8 EM → 7.5 EM; Chat-3D v2: 45.0 → 3.4; LEO: 49.4 → 14.3; Chat-Scene: 54.4 → 17.0; GPT4Scene: 60.6 → 33.1.
VRS scores are extremely low across all models: 3D-LLM 9.6%, Chat-3D v2 6.6%, LEO 14.3%, Chat-Scene 12.9%, GPT4Scene 18.2% — confirming near-complete failure on viewpoint consistency.
Only a tiny fraction of instances are answered correctly across all four rotations: e.g., GPT4Scene achieves only 0.5% on four-correct, 2.5% on three-correct.
Per question type, existence questions score highest (up to 44.2%) while direction and distance questions are lowest (5.5–15.8%), consistent with directional reasoning being the core failure mode.
3DR-FT consistently improves performance over standard SFT on Real-3DQA across both LEO and Chat-Scene baselines (quantitative table truncated in provided text).

Limitations¶

Real-3DQA is derived from SQA3D and inherits its indoor scene domain and egocentric setup; generalization to outdoor or object-centric 3D-QA datasets is not demonstrated.
The filtering procedure relies on the specific 3D-LLMs chosen ($M_A, M_B, M_C$); different model choices could yield different question subsets.
VRS augmentations are GPT-generated and may introduce hallucination or imperfect spatial consistency despite multi-stage quality control.
The 3DR-FT reweighting uses a fixed blind model $p_\phi$ that does not update during training; stale weights may become misaligned as $p_\theta$ improves.
The paper does not evaluate on fully synthetic or outdoor 3D-QA settings, leaving broader applicability open.

Relevance to Vision-Language Models¶

This paper directly challenges the evaluation validity of 3D vision-language models, a natural extension of 2D VLMs into embodied and spatial reasoning. The linguistic shortcut problem it identifies in 3D-LLMs mirrors well-known VQA bias issues in 2D VLMs (e.g., VQA-CP), but demonstrates the problem persists even in architectures explicitly designed to consume 3D point-cloud inputs. The VRS metric introduces cross-view consistency testing as a new evaluation dimension orthogonal to per-question accuracy, which is directly applicable to any spatial VLM. The 3DR-FT strategy of using a unimodal blind model to reweight multimodal training loss is a general technique that could transfer to 2D VLMs facing similar modality-shortcut problems.