Quality-Driven Curation of Remote Sensing Vision-Language Data via Learned Scoring Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Remote sensing (RS) VLMs are bottlenecked by low-quality synthetic training data, yet no automated quality-assessment framework exists for this domain. This paper introduces ScoreRS, a learned scoring model trained on RS-specific pairwise preference data, and shows that fine-tuning on just the top 30% of ScoreRS-ranked data outperforms training on the full dataset. ScoreRS further serves as a reward model for GRPO-based RL training and a best-of-N selector at test time.

Problem

RS image-text datasets cannot be web-crawled like natural-image data, so they are built via rule-based templates or flagship VLMs (GPT-4o, Gemini). Both approaches produce data with quality defects—sparse descriptions, hallucinated content, missing salient features—yet no systematic, automated quality-scoring framework exists for RS vision-language data. CLIP-score, the de-facto proxy, is misaligned with RS-specific semantic requirements.

Method

Quality dimensions. Five RS-specific text-quality axes are defined: relevance, specificity & detail, completeness, clarity & fluency, and semantic richness. A 5-point rubric is established for each.

Preference dataset construction. - Image-caption pairs (72K): Captions generated for 76K deduplicated images (from LHRS-Align, 1.15M images) by three VLMs (LHRS-Bot-Nova, Qwen2VL-7B, InternVL-2.5-8B); GPT-4o scores and ranks them to produce (T⁺, T⁻) pairs. Human–GPT-4o agreement: 92.6%. - Vision instruction pairs (26K + 33K): Aggregated from GeoChat, LHRS-Instruct, SkysenseGPT (~726K total, deduped to 112K). Close-ended questions: wrong VLM answer = T⁻, ground-truth = T⁺. Open-ended: Qwen2VL-72B judges competing answers. An additional 33K RS-domain-specific QA pairs (12 expert categories) are generated and scored by GPT-4o.

ScoreRS training (three stages): 1. Initialize Qwen2VL-7B with a linear value head; warm-up on text-only UltraFeedback data. 2. Unfreeze ViT + value head; train on image-caption preference data. 3. Full-parameter training on vision instruction preference data + RLHF-V. Optimization via Bradley-Terry pairwise preference loss.

Applications: - Data selection: score and keep top-K% of training samples. - GRPO RL reward: binary reward for close-ended; reference-based ScoreRS reward r = 1 − exp(−(fθ(ŷ) − fθ(y)) × β) for open-ended (reward = 0 if score doesn't exceed reference). - Best-of-N selector: pick highest ScoreRS-scored candidate among N sampled responses.

Key Contributions

• First large-scale RS-specific vision-language preference dataset (72K image-caption + ~59K vision-instruction pairs).
• ScoreRS: first open-source learned quality scoring model for RS VL data, built on Qwen2VL-7B with three-stage progressive training.
• Demonstration that top-30% ScoreRS-filtered data outperforms full-dataset and CLIP-score-filtered training for both CLIP and large VLM fine-tuning.
• ScoreRS as a GRPO RL reward model enabling +5% VG-DIOR / +2% LHRS-Bench gains over supervised baseline.
• ScoreRS as a BoN selector achieving >70% on LHRS-Bench for the first time.

Results

CLIP fine-tuning (RemoteCLIP, top-30%): - Average classification @1: 70.97% (ScoreRS) vs. 69.98% (CLIP-score) vs. 63.52% (full data) vs. 57.55% (no fine-tuning). - Average retrieval R@1: 27.11% (ScoreRS) vs. 26.36% (CLIP-score) vs. 25.19% (full data).

Qwen2VL-7B fine-tuning (VHM dataset): - ScoreRS-P30S60 vs. full fine-tuning: +4.08% avg. classification, +8.97% avg. RSVQA, +2.35% VG-DIOR grounding, +1.35% LHRS-Bench. - ScoreRS-P30S60 consistently outperforms equivalent CLIP-score-filtered variants across all tasks.

Comparison with RS-specific VLMs (Qwen2VL-7B-RS): - VG-DIOR grounding: 58.34% vs. VHM's 55.99%, LHRS-Bot-Nova's 31.51%. - LHRS-Bench: 67.08% vs. best competitor (InternVL-2.5-8B) at 65.86%; outperforms all RS-specific VLMs.

RL training (Qwen2VL-7B-RS-R1): - VG-DIOR: 64.52% vs. 58.34% (no RL); 62.47% without ScoreRS reward. - LHRS-Bench: 69.13% vs. 67.08% (no RL); 65.71% without ScoreRS reward.

BoN selection: ScoreRS as selector surpasses CLIP-score across all models and candidate counts; LHRS-Bench first exceeds 70% with ScoreRS + Qwen2VL-7B-RS-R1 at N=15.

Multi-stage training ablation: 92.97% reward accuracy vs. 82.09% for joint training; removing either preference dataset subset degrades accuracy on the other domain.

Limitations

• ScoreRS is initialized from Qwen2VL-7B, making it heavyweight as a scoring model; efficiency of generative scoring alternatives was not explored.
• Preference labels for image-caption pairs rely on GPT-4o as judge—still subject to frontier VLM biases, despite 92.6% human agreement.
• Open-ended instruction preferences use Qwen2VL-72B (not GPT-4o) as judge due to cost, potentially introducing inconsistencies between the two preference subsets.
• RL experiments use a relatively small 8K sample training set and evaluate on two benchmarks; broader generalization is not demonstrated.
• The VHM instruction dataset derives from standard RS benchmarks, making aggressive instruction filtering (30%) harmful—domain-specific limitations of the filtering threshold are noted but not fully resolved.
• ScoreRS's utility is demonstrated only within the RS domain; cross-domain transferability is not assessed.

Relevance to Vision-Language Models

This work directly addresses a core open problem in domain-specific VLM training: how to assess and select high-quality training data when web-scale naturally aligned corpora are unavailable. The ScoreRS framework—pairwise preference data construction, Bradley-Terry reward modeling, and multi-stage training on top of a general-purpose VLM backbone—is a transferable blueprint for other data-scarce domains. The dual use of ScoreRS as both a training-time filter and a test-time BoN selector (and RL reward) connects to emerging themes of reward modeling and test-time compute scaling in the broader VLM literature. For researchers tracking VLMs, this paper demonstrates that data quality, not quantity or architecture, is the primary performance bottleneck in specialized domains, supporting the growing consensus around quality-driven scaling laws.