Skip to content

Where Do Vision-Language Models Fail? World Scale Analysis for Image Geolocalization

🕒 Published (v1): 2026-04-17 17:09 UTC · Source: Arxiv · Venue: CVPR · link

Why this paper was selected

World-scale geolocalization analysis reveals systematic VLM perception failures; CVPR

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper benchmarks nine state-of-the-art VLMs (≤8B parameters) on zero-shot country-level image geolocalization across three large-scale datasets. It introduces the Geographic Error Reasonableness (GER) score to distinguish visually justified errors from arbitrary misclassifications, and reveals that Qwen3-VL-4B approaches supervised baselines despite no task-specific training, while exposing systematic failure modes including an inverted scaling law, urban bias, and biome-dependent difficulty.

Problem

Prior work on VLM geolocalization is fragmented and small-scale (e.g., GPT-4V on 100 images). Standard accuracy metrics treat all wrong predictions equally—confusing Cambodia with Thailand is penalized identically to guessing Canada—hiding qualitatively different failure modes. There is no unified, multi-model, multi-dataset evaluation that also characterizes why and where VLMs fail at geographic reasoning.

Method

Nine instruct-tuned VLMs from three families (InternVL2.5-{1B,4B,8B}, LLaVA-{Mistral-7B, Vicuna-7B, LLaMA3-8B}, Qwen3-VL-{2B,4B,8B}) are evaluated in two settings: unconstrained (free-form top-5 JSON prediction) and label-constrained (predictions restricted to the dataset's valid country set). Greedy decoding and no fine-tuning are used throughout.

Three evaluation datasets are used: GeoGuessr-50k (49,997 Street View images, 124 countries), CityGuessr68k (68,269 YouTube frames, 91 countries), and OSV5M (50,000-image stratified subset of 5.1M Mapillary images, 219 countries).

The multi-dimensional analysis framework includes: - Urban/Rural stratification: zero-shot classification by CLIP ViT-L/14, SigLIP, and Qwen3-VL-4B; results averaged across annotators. - Biome-level stratification: six biomes (Tropical, Arid, Temperate, Mediterranean, Tundra, Boreal); only consensus-labeled samples (all three annotators agree) are used. - Neighbor Hop Distance: shortest-path border crossings on a country adjacency graph to measure geographic proximity of errors. - GER score: for an incorrect prediction \(\hat{y}_i\) on image \(x_i\), let \(\mathcal{N}_K(x_i)\) be the \(K=5\) nearest images in CLIP/SigLIP embedding space. Define \(c(\hat{y}_i, x_i) = |\{j : y_j = \hat{y}_i, x_j \in \mathcal{N}_K(x_i)\}|\). Then: $\(\text{GER-Weak} = \frac{1}{|E|}\sum_{i \in E} \mathbf{1}[c(\hat{y}_i, x_i) \geq 1], \quad \text{GER-Strong} = \frac{1}{|E|}\sum_{i \in E} \mathbf{1}[c(\hat{y}_i, x_i) \geq 2]\)$ where \(E\) is the set of incorrect predictions. GER is averaged over CLIP and SigLIP neighbors.

Key Contributions

  • First systematic, multi-model, multi-dataset benchmark for zero-shot VLM country-level geolocalization (9 models, 3 datasets, ~168K images total).
  • GER score: a novel metric separating visually justified confusions from arbitrary misclassifications, complementing hop distance.
  • Discovery of inverted scaling in Qwen3-VL: the 4B model outperforms the 8B (e.g., +7.49 pp on GeoGuessr-50k Top-1), attributed to the language decoder rather than the shared vision encoder.
  • Characterization of a universal urban–rural performance gap validated across three independent annotators.
  • Biome-level difficulty analysis showing dataset-dependent rankings (e.g., Boreal is hardest on GeoGuessr-50k but easiest on OSV5M for top models).

Results

Overall Top-1 accuracy (unconstrained): - Qwen3-VL-4B: 74.79% / 65.78% / 45.45% (GeoGuessr / CityGuessr / OSV5M) — best on two of three datasets. - Qwen3-VL-2B leads OSV5M at 46.50%; Qwen3-VL-8B lags at 67.30% / 65.31% / 41.78%. - InternVL2.5-8B: 65.00% / 57.21% / 36.30% (conventional monotonic scaling). - LLaVA variants are weakest: LLaVA-Mistral-7B scores only 17.67% Top-1 on OSV5M.

vs. supervised baseline: - Zero-shot Qwen3-VL-4B (65.96% Top-1 on CityGuessr validation) approaches GAEA (69.6%), which is fine-tuned on CityGuessr data, and outperforms LLaVA-OneVision (55.1%) and InternVL2-8B (52.3%).

Urban–Rural gap: - Qwen3-VL-4B on GeoGuessr-50k: 83.8% urban vs. 68.5% rural (~15 pp gap). - Largest gap on CityGuessr: 68.84±0.78% urban vs. 41.54±5.21% rural. - Gap compresses on OSV5M (Qwen3-VL-2B: ~5 pp gap).

Biome difficulty (GeoGuessr-50k, Qwen3-VL-4B): - Mediterranean: 73.8% Top-1; Boreal: 52.1% Top-1. - On OSV5M the ranking inverts: Boreal best (76.8%, Qwen3-VL-2B), Tropical hardest (20.5%).

GER (unconstrained, GeoGuessr-50k): - GER-Weak ranges from ~10.7% (LLaVA-Vicuna) to ~44.4% (Qwen3-VL-4B), indicating stronger models make more visually plausible errors.

Limitations

  • All models are capped at ≤8B parameters; behavior of larger VLMs (e.g., GPT-4o, Gemini) is not evaluated.
  • GeoGuessr-50k has a heavy US bias (~20% of images) and retains game-UI overlays (compass, map widget) that may aid or confound predictions.
  • CityGuessr labels are derived from YouTube video middle-frames, which may not be geographically representative of each city.
  • OSV5M biome consensus subset is only 36% of images; results may not generalize to the full distribution.
  • GER relies on CLIP/SigLIP visual neighbors as a proxy for "visual similarity," which may not capture all culturally relevant cues.
  • LoRA fine-tuning results are described as "preliminary" with limited detail in the excerpt provided.
  • Country-level granularity is coarse; fine-grained localization (city, GPS) is out of scope.

Relevance to Vision-Language Models

This paper directly probes an underexplored capability of VLMs—implicit geographic world knowledge—offering a diagnostic framework that goes beyond accuracy to explain failure modes through GER and hop distance. The inverted scaling finding in Qwen3-VL challenges the assumption that larger language decoders uniformly improve vision-grounded reasoning, with direct implications for VLM architecture choices. The urban bias and biome-dependent difficulty expose systematic gaps in how VLMs encode environmental and cultural cues from image-text pretraining, relevant to any application requiring geographic or scene-level semantic understanding. The zero-shot performance approaching fine-tuned supervised models underscores that modern VLMs encode rich geospatial priors, making this benchmark a useful probe for future multimodal foundation models.