Skip to content

Towards Faithful Reasoning in Remote Sensing: A Perceptually-Grounded GeoSpatial Chain-of-Thought for Vision-Language Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; perceptually grounded chain-of-thought for faithful VLM geospatial reasoning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RSThinker introduces Geo-CoT, a perceptually-grounded chain-of-thought framework for remote sensing VLMs that mandates explicit spatial references at each reasoning step. Trained via a two-stage SFT→GRPO pipeline on the new Geo-CoT380k dataset (384,591 structured rationales), it produces verifiable, bounding-box-linked reasoning traces rather than opaque pixel-to-answer mappings. It achieves state-of-the-art performance across visual grounding, object counting, detection, classification, VQA, and captioning benchmarks.

Problem

Existing remote sensing VLMs learn an end-to-end mapping from pixels to final output, treating intermediate reasoning as a latent variable. This produces plausible but unverifiable outputs prone to hallucination—a critical failure mode for high-stakes tasks like disaster response. Prior MM-CoT efforts either reason semantically (relying on parametric world knowledge rather than visual evidence) or include non-localizable textual references that cannot be audited against specific pixel regions. No large-scale dataset existed to train truly perceptually-grounded reasoning in Earth Observation VLMs.

Method

Geo-CoT cognitive architecture: Each reasoning step must be linked to a verifiable spatial reference (bounding box coordinates), following a three-phase protocol—Planning (systematic search strategy), Grounding (iterative evidence gathering with explicit pixel references), and Synthesis (final answer from accumulated evidence). Outputs are structured as <think>...</think><answer>...</answer>.

Geo-CoT380k dataset construction: GPT-4V generates structured rationales conditioned on verified bounding boxes, image captions, and CoT exemplars—not open-ended generation—covering 384,591 samples across VQA, captioning, scene classification, visual grounding, object counting, and detection (Table 1).

Two-stage alignment: 1. Stage I — SFT: Fine-tunes GLM-4.1V-9B-Base using standard autoregressive loss \(\mathcal{L}_\text{SFT}(\theta) = -\sum_{t=1}^{|o_i|} \log p(o_{i,t}|o_{i,<t}, I, Q; \theta)\) to instill the Geo-CoT cognitive structure. 2. Stage II — GRPO: Samples \(k\) outputs per input, normalizes rewards group-wise to compute advantage \(\hat{A}_i = \frac{R_i - \text{mean}(R)}{\text{std}(R)}\), and optimizes a clipped surrogate with KL regularization toward the SFT reference policy. Task-specific rewards directly implement canonical metrics (IoU for grounding, MAE-derived accuracy for counting, mAP@0.5 for detection, weighted captioning metrics).

Base model is GLM-4.1V-9B-Base with Aimv2-Huge ViT and dynamic positional encoding via bicubic interpolation: \(P_\text{adapted}(g) = I_\text{bicubic}(P_\text{orig}, g_\text{norm})\).

Key Contributions

  • Formal definition of Perceptually-Grounded Geo-CoT: a reasoning paradigm requiring explicit spatial anchoring of every claim to pixel-level evidence via bounding box coordinates.
  • Geo-CoT380k: first large-scale RS CoT SFT dataset (384,591 samples) spanning six task types from diverse public benchmarks.
  • RSThinker: a 9B VLM trained via SFT→GRPO showing the two-stage ordering (SFT as prerequisite for GRPO) is essential for eliciting faithful structured reasoning.
  • State-of-the-art across a comprehensive RS benchmark suite covering the full spectrum from object-level to scene-level tasks.

Results

Visual Grounding (Table 4): - VRSBench-VG @0.5: 90.4 vs. SkySenseGPT 63.5, GLM-4.1V-Thinking 63.8 - DIOR-RSVG @0.5: 94.0 vs. EarthDial 72.5 - RSVG (zero-shot) @0.5: 64.0 vs. EarthDial 42.0 - RRSIS-D (zero-shot) @0.5: 93.1 vs. EarthDial 42.0

Object Counting (Table 5): - DOTAv2-val Acc: 43.93 vs. SkySenseGPT 33.11 - HRRSD Acc: 85.26 vs. EarthDial 61.45 - RSOD (zero-shot) Acc: 80.0 vs. ChatGPT-5 40.0 - NWPU-VHR (zero-shot) Acc: 46.5 vs. GLM-4.1V-Thinking 28.5 (MAE 0.242 vs. 1.194)

Scene Classification (Table 6): - RESISC45: 98.17, AID: 99.74, WHU-RS19: 96.89, WHU-SIRI: 92.14

VQA (VRSBench-VQA): 66.95 vs. EarthDial 64.94

Image Captioning: Outperforms SOTA on NWPU, RSICD, RSITMD, UCM, Sydney, VRSBench captioning (BLEU-4)

Limitations

  • Geo-CoT380k rationales are generated by GPT-4V, inheriting any systematic biases or hallucinations of that model despite conditioning on ground-truth annotations.
  • The approach is evaluated on standard RS benchmarks; generalization to unseen sensor modalities (SAR, hyperspectral) or truly novel geospatial query types is not demonstrated.
  • Computational cost of generating 384,591 GPT-4V rationales and two-stage training is not discussed in terms of feasibility for community replication at scale.
  • Zero-shot evaluations (RSVG, RSOD, NWPU-VHR, RRSIS-D) are limited; in-domain fine-tuning on those sets was not performed, leaving headroom analysis unclear.
  • The paper text is truncated before ablation studies and qualitative analysis, so component-level evidence for why SFT→GRPO ordering is necessary is not fully visible here.

Relevance to Vision-Language Models

This paper is a direct advance on grounded multimodal reasoning, extending the Visual CoT / VoCoT line of work to a domain (remote sensing) where salient-object assumptions of prior grounded CoT frameworks break down. The SFT→GRPO two-stage recipe mirrors the DeepSeek-R1 paradigm and demonstrates how outcome-based RL with domain-specific reward shaping can steer a VLM toward factually faithful reasoning without human-annotated reasoning traces. For VLM researchers, the key insight is that perceptual grounding (bounding-box-anchored evidence) is a necessary precondition—not a byproduct—of reliable CoT in visually complex domains, challenging designs that treat chain-of-thought as purely linguistic. Geo-CoT380k also provides a scalable template for constructing task-specific CoT corpora via conditioned expert-model annotation.