Recognition through Reasoning: Reinforcing Image Geo-localization with Large Vision-Language Models¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

GLOBE reframes image geo-localization as an interpretable reasoning task by fine-tuning a 7B LVLM with Group Relative Policy Optimization (GRPO) over three task-specific rewards. It introduces MP16-Reason, a 33K-sample dataset distilled from diverse social media images with localizability labels, reasoning trajectories, and multi-model verification. GLOBE outperforms open-source LVLMs up to 72B parameters on multiple geo-localization benchmarks while requiring far less training data than prior methods.

Problem¶

Existing geo-localization methods are either classification/retrieval pipelines that lack interpretability, or LVLM-based approaches that rely on supervised fine-tuning (SFT) on street-view-dominated datasets — yielding shallow pattern replication rather than genuine visual-geographic reasoning. No existing dataset combines scene diversity (beyond street view) with explicit, verified reasoning supervision.

Method¶

Dataset (MP16-Reason): Raw MP-16 social media images are processed through three strong VLMs (Qwen2.5-VL-72B, InternVL3-78B, GeoCLIP) via knowledge distillation to produce localizability scores, step-by-step reasoning trajectories, and location predictions. Multi-dimensional verification filters outputs via: (1) threshold-based localizability filtering, (2) cross-model self-verification (city/country-level agreement + semantic reasoning alignment), and (3) visual-semantic consistency using a semantic segmentation model to check that entities mentioned in reasoning actually appear in the image. This yields 33K clean training samples.

GRPO Fine-tuning (GLOBE): A Qwen2.5-VL-7B backbone is fine-tuned directly with GRPO (no SFT cold start) using three composite rewards: - Localizability reward \(R_\text{loc}\): LLM-based reward model scoring the probability that image–reasoning pair is localizable. - Visual grounding consistency reward \(R_\text{vis}\): soft entity-matching ratio between reasoning-mentioned entities and segmentation-detected visual elements. - Geo-localization accuracy reward \(R_\text{geo}\): hierarchical classification reward — 0 for wrong country, partial \((1-\alpha)\) for correct country/wrong city, 1 for both correct.

The composite reward \(r = \lambda_1 R_\text{loc} + \lambda_2 R_\text{vis} + \lambda_3 R_\text{geo}\) drives GRPO's group-normalized advantage optimization with a KL penalty against the reference policy.

Key Contributions¶

MP16-Reason dataset: 33K training + 12K test samples from social media imagery (134/145 countries, 1944/3012 cities) with multi-model distilled and verified reasoning supervision.
Three-reward GRPO framework: jointly optimizes localizability, visual grounding, and geo-accuracy in a single RL training loop — no SFT cold start needed.
GLOBE-7B: open-source LVLM that outperforms 72B open-source models on geo-localization benchmarks despite 100× fewer training samples than image-only baselines.

Results¶

On MP16-Reason-Test (GLOBE-7B vs. best open-source baselines): - City-level (25 km): 62.85% vs. Qwen2.5-VL-72B 59.30%, GeoReasoner-7B 40.44% - Country-level (750 km): 86.68% vs. Qwen2.5-VL-72B 84.06%, GeoReasoner-7B 68.01% - Continent (2500 km): 92.52% vs. Qwen2.5-VL-72B 91.65%

On IM2GPS3K: - City (25 km): 40.18% vs. GeoCLIP 34.47%, Qwen2.5-VL-72B 35.77%, GeoReasoner-7B 26.94% - Country (750 km): 71.45% vs. GeoCLIP 69.67%, Qwen2.5-VL-72B 64.96%

Ablation (reward components): Removing any single reward causes noticeable drops; full 3-reward GRPO at 62.85% city vs. SFT 56.76%, baseline 52.72%.

Ablation (data curation): Full multi-source validation outperforms random sampling (62.85% vs. 59.22% city-level GRPO) and single-source validation (59.45%).

GLOBE remains behind closed-source systems (GPT-4.1: 66.76% city on MP16-Reason-Test, Doubao1.5-VL: 64.02%).

Limitations¶

Fine-grained coordinate-level localization is weak — semantic reasoning cues (architecture, vegetation) cannot distinguish between nearby locations sharing similar visual patterns (e.g., multiple Mediterranean cities).
Performance gap remains vs. proprietary closed-source models (GPT-4.1, Doubao1.5-VL) whose training data and scale are undisclosed.
Azure Maps coordinate lookup introduces quantization error (returns geographic center of predicted city/country) that penalizes the 1 km street-level metric.
Training throughput is low (~0.44 examples/second on 8× H20 GPUs), limiting scalability of GRPO to larger datasets.
Generalization to street-view domains (OSV-5M) shows improvement but remains below retrieval-specialized baselines at fine-grained thresholds per appendix results.

Relevance to Vision-Language Models¶

This paper directly advances LVLM adaptation via reinforcement learning, demonstrating that GRPO with structured, task-specific rewards substantially outperforms SFT for open-ended visual reasoning tasks — a finding applicable beyond geo-localization to any task requiring grounded multimodal inference. The multi-model distillation + cross-model verification pipeline for generating reasoning supervision is a practically useful recipe for curating LVLM training data in domains where ground-truth reasoning chains are unavailable. The visual grounding consistency reward, which penalizes hallucinated entities through semantic segmentation cross-checking, is a novel mechanism for reducing LVLM hallucination during RL training. For researchers tracking VLMs, GLOBE exemplifies the "reasoning via RL" paradigm (akin to DeepSeek-R1 for text) being extended to spatial/geographic understanding.