Skip to content

DM-KG: A Novel Method for Boosting Spatial Cognition of Vision-Language Models in Street View Imagery

🕒 Published (v1): 2026-07-14 03:52 UTC · Source: Arxiv · link

Why this paper was selected

Spatial cognition augmentation for VLMs on street-view; directly relevant to geo-VLM eval

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DM-KG augments VLMs with an explicit Direction-Metric Knowledge Graph built from a single street view image by combining panoptic segmentation with monocular metric depth estimation. The resulting JSON-formatted spatial prior is injected into VLM prompts, converting hallucination-prone visual guessing into deterministic geometric reasoning. On SpatialRGPT-Bench, DM-KG cuts distance MAE by 31.1% and mean angular error by 65.8% relative to baseline VLMs.

Problem

Existing VLMs suffer from "spatial semantic hallucinations" when answering metric spatial questions (object distances, directions) in street view imagery. Data-driven approaches (e.g., SpatialVLM) exploit linguistic/statistical biases rather than true geometric reasoning, while architecture-enhanced models (e.g., SpatialRGPT) remain opaque black boxes whose direction and distance errors cannot be traced or calibrated—blocking deployment in GeoVQA, visual localization, and navigation tasks.

Method

DM-KG is a four-stage inference-time pipeline requiring no VLM fine-tuning:

  1. Stage 1 — 2D Scene Understanding. Mask2Former panoptic segmentation produces pixel-level instance masks \(M_i\) with category labels \(c_i\) for all scene entities.

  2. Stage 2 — 3D Metric Recovery. A monocular metric depth estimator (Metric3Dv2-g recommended) produces a depth map \(D \in \mathbb{R}^{H \times W}\). For each entity, depths within \([P_{10}, P_{90}]\) of the mask are retained and the median \(d_i\) is used. The mask centroid \((u_i, v_i)\) is back-projected to camera-frame 3D coordinates via the pinhole model: $\(x = \frac{(u - c_x) \cdot z}{f_x}, \quad y = \frac{(v - c_y) \cdot z}{f_y}, \quad z = d_i\)$ Only the horizontal \((x, z)\) plane is used for downstream spatial reasoning.

  3. Stage 3 — DM-KG Construction. A graph \(G = \langle E, R \rangle\) is assembled in JSON. Entity nodes encode (id, category, 2D bbox, 3D centroid); relational edges encode bidirectional azimuth \(\theta_{A \to B} = \arctan2(x_B - x_A, z_B - z_A)\) (quantized to clock hours), semantic direction strings, and Euclidean distance \(d_{A,B} = \sqrt{(x_B-x_A)^2 + (z_B-z_A)^2}\). Only entity pairs within 40 m of camera and 30 m of each other are included to suppress noise.

  4. Stage 4 — Prompted VLM Reasoning. Entity masks are overlaid on the image with color highlights for visual grounding. The JSON graph plus Chain-of-Thought instructions are injected as a structured prompt, shifting the VLM's role from spatial estimator to symbolic retriever performing lookup-and-arithmetic reasoning.

Key Contributions

  • A training-free, model-agnostic spatial reasoning enhancement framework that injects explicit geometric priors rather than relying on implicit learned spatial statistics.
  • A joint panoptic segmentation + metric depth pipeline for constructing entity-level 3D knowledge graphs with semantic attributes and geometric topological constraints.
  • A standardized benchmark evaluating four mainstream monocular depth estimation models (Metric3Dv2, DepthAnythingV2, DepthPro, UniDepthV2) across three distance intervals (5–10 m, 10–20 m, >20 m) on KITTI, providing a model-selection guide for street-view 3D coordinate tasks.
  • A visual-text bidirectional anchoring mechanism (colored mask overlays + bounding box JSON) that resolves entity-symbol alignment failures during VLM spatial QA.

Results

Evaluated on SpatialRGPT-Bench (200+ QA images from KITTI and NuScenes) against GPT-5.4 (baseline) and SpatialRGPT (spatial-specialized model):

  • Distance estimation MAE: reduced by 31.1% over baseline.
  • Direction judgment MAE: reduced from 7.71° to 3.27° (65.8% reduction); AbsRel from 0.256 to 0.109 (65.8% drop).
  • Direction judgment success rate: 95.3% (DM-KG on GPT-5.4) vs. 96.7% (SpatialRGPT)—within 1.4 pp while substantially outperforming on error magnitude.
  • Best depth model: Metric3Dv2-g with focal length input achieves overall MAE of 0.82 m (AbsRel 0.040) across all distances; far-range (>20 m) MAE is 1.73 m (AbsRel 0.046), the lowest among all evaluated models.
  • Large depth models without focal length input paradoxically underperform smaller models; providing focal length priors reverses this, yielding clear accuracy gains especially at 5–25 m.

Limitations

  • Requires known camera intrinsics for optimal depth accuracy; performance degrades without focal length priors, particularly at far range (>20 m).
  • Far-distance depth estimation (>20 m) remains the primary error bottleneck regardless of model choice.
  • The direction judgment success rate (95.3%) marginally trails SpatialRGPT (96.7%), suggesting the KG approach may still miss edge cases where architectural region features are beneficial.
  • The 40 m / 30 m inclusion thresholds are heuristic and may exclude relevant entity pairs in wide open scenes or cut off relationships prematurely in dense urban settings.
  • Paper text is truncated; full distance-estimation success-rate comparison figures are not available for review.
  • Evaluation is limited to street view imagery (KITTI, NuScenes, Google Street View); generalization to other geospatial domains (aerial, indoor) is undemonstrated.
  • DM-KG construction adds inference latency from running panoptic segmentation and depth estimation in parallel before the VLM call.

Relevance to Vision-Language Models

DM-KG is directly relevant to the ongoing effort to ground VLMs in metric physical space without expensive fine-tuning—a key open problem in geospatial and embodied AI applications. It demonstrates that decoupling geometric measurement (handled by deterministic algorithms) from symbolic reasoning (handled by the frozen VLM) substantially reduces spatial hallucinations, offering a complementary path to SpatialVLM-style dataset scaling and SpatialRGPT-style architectural augmentation. The prompt-injection framework is model-agnostic, meaning it can be wrapped around any capable frontier VLM (demonstrated on GPT-5.4), making it practically useful for researchers deploying VLMs in GeoVQA, navigation, or scene understanding pipelines. The explicit, JSON-structured knowledge graph also provides interpretability and error traceability that purely end-to-end trained spatial VLMs currently lack.