More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe

🕒 Published (v1): 2026-07-17 13:25 UTC · Source: Arxiv · link

Why this paper was selected

Luc Van Gool (ETH); simple recipe beats RS-specific archs; transferable scaling insight

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MLRS (More with Less Remote Senser) adapts a general-purpose VLM (InternVL3.5-8B) to remote sensing without any architectural modification, using large-scale multi-task reinforcement learning over 80k curated samples drawn from a 2.3M-sample pool. The model couples the VLM with SAM3 as an external localization tool, communicating entirely through text. It matches or surpasses architecturally specialized RS-VLMs on a broad suite of benchmarks, supporting the claim that data diversity and scale matter more than domain-specific design.

Problem

Most remote sensing VLMs address domain-specific challenges (small objects, SAR modality, ultra-high-resolution, multi-temporal/multi-view inputs) by adding specialized encoders, alignment modules, modality-fusion branches, or task-specific segmentation heads. This creates a fragmented landscape where every new capability requires a new architecture. The paper challenges whether such specialization is necessary, or whether a strong general-purpose VLM trained at sufficient scale can achieve the same outcome.

Method

Architecture: MLRS uses InternVL3.5-8B unchanged as the central language policy. For localization tasks, the VLM emits a structured text-based tool call containing a noun phrase and bounding boxes; these are passed to SAM3, whose output masks are filtered through the predicted boxes. For language tasks (VQA, captioning), the VLM answers directly. No task routers, modality encoders, or fusion modules are added.

Training objective (GRTO): The VLM is optimized with GRPO; a differentiable surrogate loss (binary cross-entropy + semantic IoU) is applied to SAM3 simultaneously, enabling end-to-end joint training. This makes MLRS the first RS-VLM to combine a trained dense-mask segmentation head with RL post-training.

Adaptive reward: \(R = \lambda_{\mathrm{fmt}} F + \lambda_{\mathrm{task}} S\) with \(\lambda_{\mathrm{fmt}} = 0.1\), \(\lambda_{\mathrm{task}} = 0.9\). The task score \(S\) varies by output type: - MC-VQA: set IoU between predicted and target option sets - Free-form VQA / captioning: cosine similarity via Qwen3-Embedding-0.6B - Segmentation: semantic mask IoU - Detection: mean matched box IoU at greedy assignment

Data: 80k samples balanced across tasks and difficulty levels from a 2.3M raw pool; domain coverage deliberately varies in source count (1 source for multi-view, 13 for segmentation) to measure how per-domain diversity drives OOD generalization. Training uses LoRA (rank 64), batch size 16 (128 rollouts with group size 8), 5k steps on 8 × H200 GPUs.

Key Contributions

• MLRS model: a competitive RS-VLM spanning VQA, captioning, detection, grounding, and segmentation over optical, SAR, ultra-high-resolution, multi-temporal, multi-modal, and multi-view inputs—without any new architectural components.
• First combination of trained segmentation head + RL post-training in RS-VLMs, via GRTO enabling joint VLM+SAM3 optimization.
• Empirical data-scaling study: shows that in-distribution and OOD performance correlate with per-domain source diversity, not raw data volume alone; multi-task RL improves generalization across most domains as training scales.
• Broadest task/input-type coverage among published RS-VLMs (see Table 1).

Results

All comparisons are against the untuned base model (InternVL3.5-8B) or named specialist baselines at ~8B scale; zero-shot OOD benchmarks highlighted.

• SAR captioning (SARLANG-1M, G-Eval): \(0.09 \to 0.38\) (+322% over base)
• Referring segmentation (EarthReason, IoU): \(0.27 \to 0.68\); (LaSeRS, gIoU): \(0.31 \to 0.67\)
• OOD detection precision (GEOBench-VLM, Pr@0.5): \(0.24 \to 0.56\) (vs. base); absolute score 55.74 vs. EarthDial's 24.29
• OOD detection (XLRS-Bench, Acc@0.5): \(0.054 \to 0.409\) — a \(7.6\times\) gain
• OOD segmentation (GeoSeg-Bench2, IoU): \(0.41 \to 0.70\); (GEOBench-VLM): \(0.08 \to 0.16\)
• OOD MC-VQA (GEOBench-VLM): \(39.41 \to 50.28\); (XLRS-Bench): \(39.04 \to 51.34\)
• SAR GEO captioning (GEOBench-VLM, G-Eval rescaled): \(37.89 \to 86.88\)
• Multi-view VQA (UrBench): marginal gain \(0.498 \to 0.519\), plateaus at 3k steps
• Temporal VQA: peaks at 1k steps then degrades; ends below base on VLRS-Bench (\(0.274 \to 0.230\)) and RSHR-Bench (\(0.345 \to 0.295\))

Limitations

• Multi-view generalization is weak: single training source (CVG-Text) limits OOD transfer; UrBench improvement is minimal.
• Temporal VQA degrades: OOD temporal performance falls below the base model after 1k steps; the mechanism is unclear but may reflect conflicting gradient signals or insufficient source diversity (only 2 temporal sources).
• OOD captioning remains flat: training does not transfer to held-out captioning benchmarks, likely because captioning reward (embedding similarity) provides a weak learning signal relative to classification/localization tasks.
• Polygon-free segmentation requires SAM3: the tool-call interface depends on SAM3 availability; the model cannot produce spatial outputs in environments where only pure-language output is feasible.
• LoRA-constrained capacity: all runs use LoRA rank 64, potentially limiting adaptation depth for highly specialized modalities.

Relevance to Vision-Language Models

This paper directly challenges the dominant paradigm in domain-adapted VLMs that architectural specialization is necessary for strong performance, providing a clean empirical counterargument: a frozen-architecture general VLM fine-tuned with diverse multi-task RL can match or beat specialized designs. The GRTO framework — jointly optimizing a VLM and an external tool via surrogate gradients — generalizes beyond remote sensing and is relevant to any VLM-plus-tool system (e.g., referring segmentation, embodied agents). The finding that OOD generalization tracks per-domain source diversity rather than data volume shifts attention toward dataset construction methodology. For VLM researchers, this is a concrete data point supporting the "data-centric scaling" hypothesis over "architecture-centric" approaches.