Skip to content

GeoLLaVA-8K: Scaling Remote-Sensing Multimodal Large Language Models to 8K Resolution

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GeoLLaVA-8K is the first remote-sensing (RS) multimodal large language model capable of processing inputs up to 8K×8K resolution. It addresses the dual bottlenecks of UHR RS MLLM development—scarce training data and token explosion—through two new datasets and two token compression strategies. Trained on these resources, it achieves state-of-the-art on XLRS-Bench, outperforming models up to 72B parameters with only 7B.

Problem

Existing MLLMs are limited to ≤4K resolution and achieve <45% accuracy on XLRS-Bench (average 8,500×8,500 images). Two root causes: (1) no UHR RS image-text training datasets exist (prior RS datasets average <1K×1K resolution); (2) 8K×8K inputs produce tens of thousands of visual tokens, causing out-of-memory (OOM) errors and low semantic density—background pixels (ocean, forest, fields) constitute up to 73.14% of RS image area, contributing semantically empty tokens that dilute object-centric information.

Method

Data: Two new datasets are introduced. SuperRS-VQA (12,228 samples, avg. 8,376Ă—8,376) is fully manually annotated by RS/MLLM experts over 40 days across 22 sub-tasks. HighRS-VQA (69,139 samples, avg. 2,000Ă—1,912) uses a semi-automated pipeline: GPT-4o generates QA pairs from existing detection/segmentation annotations, then an influence-based data selection (InfSGD, adapted from LESS) retains the top-70% most influential samples by cosine similarity of LoRA-projected gradients between training candidates and a UHR validation set.

Token compression (two steps, applied between visual encoder and projection layer): 1. Background Token Pruning (BTP): Iterative local neighborhood clustering assigns each token to its most semantically similar neighbor; tokens belonging to homogeneous background clusters (high similarity to background term embeddings like "ocean," "desert") are merged and pruned. 2. Anchored Token Selection (ATS): After BTP, CLS-to-patch attention scores from the ViT's second-to-last layer rank the remaining tokens; top-R tokens (by compression ratio r) are retained, ensuring sparse foreground objects survive pruning.

The model is initialized from LLaVA-Next-2K pretrained weights and SFT-tuned on the two new datasets.

Key Contributions

  • SuperRS-VQA and HighRS-VQA: Highest-resolution RS vision-language training datasets to date; 81,367 total QA pairs across 22 sub-tasks spanning perception and reasoning.
  • Empirical characterization of UHR RS token redundancy: Logit lens and ablation experiments quantifying that (a) background covers 73.14% of RS images and degrades MLLM performance, and (b) object tokens are sparse but encode disproportionate task-relevant information.
  • Background Token Pruning: Adaptive iterative clustering of semantically similar background tokens to compress redundant visual sequences.
  • Anchored Token Selection: CLS-attention-guided retention of foreground tokens post-pruning to prevent loss of small objects.
  • GeoLLaVA-8K: First RS MLLM processing genuine 8KĂ—8K input; SOTA on XLRS-Bench with 7B parameters.

Results

  • GeoLLaVA-8K achieves 51.5% average accuracy on XLRS-Bench vs. next-best open-source Qwen2.5-VL-72B at 50.2% and Gemini 2.0 Flash at 48.7% (closed-source leader).
  • Outperforms GeoChat (22.9%), GPT-4o (32.4%), and Claude 3.7 Sonnet (40.5%).
  • Ablation: high-resolution data alone raises baseline from 44.7% → 49.1%; adding BTP+ATS reaches 51.5% (BTP alone causes OOM at 8K without ATS).
  • Small-object subset (XLRS-Bench, <5% area): GeoLLaVA-8K 40.3% vs. LLaVA-Next 34.1% overall; spatial relationship 35.0% vs. 20.0%, counting 32.5% vs. 22.5%.
  • Generalization on LRS-VQA: 56.28% vs. LLaVA-Next 55.07% (+1.21%).
  • At 24Ă— compression: 14K visual tokens, 198.06 TFLOPs, 2.17 s/image; at 96Ă—: 3.6K tokens, 51.13 TFLOPs, 1.46 s/image.

Limitations

  • Evaluated only on optical satellite imagery; performance on other modalities (SAR, multispectral) is untested.
  • CLS attention occasionally concentrates on a single dominant region rather than balancing multiple small objects, leaving multi-object focus as an open problem.
  • SuperRS-VQA is small (12K samples) due to high annotation cost; scaling manual UHR annotation remains impractical without further automation.
  • Generalization gains on out-of-distribution benchmarks (LRS-VQA) are modest (+1.21%), and conversion to MCQ format introduces potential annotation artifacts.
  • HighRS-VQA generation required >$1,000 in GPT-4o API costs and still required human quality-checking, limiting reproducibility at scale.

Relevance to Vision-Language Models

This work directly advances domain-specific VLM scaling by demonstrating that resolution limits in current MLLMs (GPT-4o, Qwen2.5-VL, InternVL) are not fundamental—they stem from data and token-management gaps addressable with targeted engineering. The Background Token Pruning and Anchored Token Selection strategies are generally applicable to any VLM processing imagery with spatially sparse semantic content, making them relevant beyond RS. The influence-based data selection pipeline (InfSGD) also offers a principled alternative to heuristic SFT data curation, applicable to VLM fine-tuning broadly. For researchers tracking VLMs, this is a strong case study in domain adaptation at extreme resolution with sub-100B parameter efficiency gains.