Skip to content

MonoIR-RS: Infrared Remote Sensing Vision-Language Learning with CLIP and VLM Adaptation

🕒 Published (v1): 2026-07-07 17:53 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MonoIR-RS introduces a large-scale infrared remote-sensing vision-language dataset (600K synthetic IR images; 59,032 IR-aware caption records) and benchmark that adapts CLIP and VLM backbones to reason from grayscale structure and thermal contrast rather than RGB color. The infrared images are synthesized from visible remote-sensing sources via DiffV2IR diffusion and paired with Qwen2.5VL-72B-rewritten captions that suppress color language. IR-aware fine-tuning lifts CLIP mean recall by up to +12.8 points and drives VLM IR-cue coverage to 100% while reducing residual RGB-leakage to near zero.

Problem

Existing remote-sensing vision-language resources and models—including RemoteCLIP, GeoRSCLIP, and RS5M—are dominated by visible-band imagery, so their captions describe color cues that are absent or misleading in infrared scenes. Infrared-specific VLM work (Infrared-LLaVA, IRGPT) targets general surveillance IR, not remote sensing, and real sensor captures are too scarce to scale. There is no large-scale, split-clean, IR-only benchmark that isolates whether IR-aware supervision actually improves alignment rather than RGB-biased baselines.

Method

Data construction. MonoIR-RS reuses the FusionRS source pool (RS5M, SkyScript, NWPU, RSICD, RSITMD) but removes the RGB modality from model-facing inputs. Infrared images are generated from registered visible images using DiffV2IR, a progressive full-range-to-target-wavelength diffusion model. IR-aware captions are rewritten by Qwen2.5VL-72B-Instruct to replace RGB-dependent terms (e.g., "green field") with grayscale structure and thermal-contrast descriptions. Strict split hygiene is enforced: train/val/test boundaries are shared across both model families; 280 RGB-named paths are removed from the 10K test split, leaving 9,720 evaluation samples.

CLIP adaptation. Five backbones (OpenAI ViT-B/32, ViT-L/14, OpenCLIP ViT-B/32, RemoteCLIP ViT-B/32, GeoRSCLIP ViT-B/32) are contrastively fine-tuned on (IR image, IR-aware caption) pairs. The text encoder is frozen; only the visual tower and projection layers are updated. Checkpoints are selected by validation mean recall.

VLM adaptation. Six backbones (Qwen2.5-VL-7B, InstructBLIP-FLAN-T5-XL, LLaVA-1.5-7B, LLaVA-1.6-Vicuna-7B, GeoChat-7B, H2RSVLM-VHM-7B) are instruction-tuned with LoRA (\(r=32\), \(\alpha=64\), dropout 0.05) plus projector tuning where exposed; the base LLM is frozen. A 32,000-conversation multitask subset (captioning, scene classification, object presence, IR-cue explanation, visual-evidence QA, hard-negative reasoning) is drawn from the 48,616 train records.

Key Contributions

  • MonoIR-RS dataset: 600K synthetic IR images + 59,032 IR-aware caption records with unified train/val/test split and quality audits (split-overlap, RGB-leakage, missing-image checks).
  • Validated synthetic modality: DiffV2IR IR is closer to real aerial IR (AVIID) than grayscale conversion by FID (85.2 vs. 126.3) and all four histogram metrics (KL, JS, \(\chi^2\), histogram intersection).
  • IR-CLIP benchmark: bidirectional retrieval evaluation on 9,720-image filtered test split across five backbones, calibrated against zero-shot baselines, with seed-stability check on ViT-B/32.
  • IR-VLM diagnostic suite: five complementary metrics (IR-cue rate, color-leakage rate, overclaim rate, class-hit rate, response length) across six VLM backbones.
  • Transfer sanity check on AVIID: IR-CLIP checkpoint retrieves visible targets from real-IR queries at 18.7% mean recall (+6.5 over zero-shot), demonstrating cross-domain representation transfer.

Results

  • IR-CLIP retrieval (9,720-image filtered test split, bidirectional mean recall mR):
  • OpenAI CLIP ViT-L/14: mR 19.2% (zero-shot 6.3%, +12.8)
  • GeoRSCLIP ViT-B/32: mR 13.4% (zero-shot 4.2%, +9.2)
  • OpenCLIP ViT-B/32: mR 12.8% (zero-shot 5.5%, +7.3)
  • OpenAI CLIP ViT-B/32: mR 12.0% (zero-shot 4.3%, +7.7)
  • RemoteCLIP ViT-B/32: mR 4.3% (zero-shot 0.8%, +3.5)
  • VLM captioning (six backbones, aggregate): IR-cue coverage reaches 100%; RGB-color leakage reduced to near zero vs. zero-shot baselines.
  • AVIID transfer (auxiliary): Real-IR-query → visible-target retrieval: 18.7% mR (+6.5 vs. zero-shot 12.2%); synthetic-IR-query → real-IR-target: +23.4 points.
  • Modality validation (AVIID, 804 test images): FID 85.2 (synthetic IR) vs. 126.3 (grayscale); histogram intersection 0.822 vs. 0.711.

Limitations

  • Infrared images are synthesized, not sensor-captured; DiffV2IR output is an infrared-like approximation, not radiometrically calibrated, and an IRGPT-style argument that synthetic IR introduces a modality gap applies here.
  • All CLIP and VLM experiments are single-run (except a three-seed check on ViT-B/32), so variance estimates are limited.
  • Absolute retrieval recall remains low even after adaptation (best mR 19.2% on a 9,720-candidate pool), suggesting large headroom and possibly insufficient scale or backbone capacity.
  • IR-aware captions are generated by a single LLM (Qwen2.5VL-72B-Instruct); caption quality and diversity are not independently verified against human annotations.
  • VLM evaluation is diagnostic (metric-based) rather than human-judged; the five metrics may not fully capture semantic faithfulness.
  • The benchmark is limited to remote-sensing scene categories from five existing RGB datasets, inheriting their class and geographic distribution biases.

Relevance to Vision-Language Models

MonoIR-RS directly addresses the modality-coverage gap in remote-sensing VLMs by constructing the first large-scale, split-clean IR-only vision-language resource and showing that IR-aware supervision measurably realigns CLIP embedding spaces and VLM generation behavior away from RGB priors. For researchers tracking VLMs, the five-diagnostic-metric framework (IR-cue rate, color-leakage rate, overclaim rate, class-hit rate, response length) offers a reusable protocol for auditing modality-specific grounding failures beyond aggregate captioning scores. The LoRA-based adaptation study across six heterogeneous VLM backbones under a shared data boundary provides a controlled comparison of how existing multimodal instruction-following priors transfer to a non-RGB domain, relevant to broader work on domain-specific VLM adaptation. The dataset and protocol also serve as a reference for future work on spectral-modality extension of foundation models (thermal, SAR, multispectral) where color-language entanglement is a systematic failure mode.