Skip to content

InstructSAM: A Training-free Framework for Instruction-Oriented Remote Sensing Object Recognition

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

InstructSAM is a training-free framework for instruction-oriented object counting, detection, and segmentation (InstructCDS) in remote sensing imagery. It chains an LVLM (for counting and category prediction), SAM2 (for class-agnostic mask proposals), and a domain-specific CLIP (for semantic similarity) into a Binary Integer Programming assignment step that eliminates confidence-threshold tuning. The authors also introduce EarthInstruct, the first benchmark for InstructCDS across open-vocabulary, open-ended, and open-subclass scenarios.

Problem

Existing open-vocabulary and visual grounding methods for remote sensing require explicit category lists, cannot handle implicit or complex reasoning queries, and rely on fragile confidence thresholds for filtering. Specialized RS models are trained on datasets with only dozens of categories, limiting zero-shot generalization. Inference time for generative detectors (e.g., Qwen2.5-VL) scales linearly with object count, making them impractical for dense scenes.

Method

InstructSAM decomposes InstructCDS into three parallel/sequential steps: 1. LVLM Counter: Given an image and a structured JSON prompt (encoding dataset-specific annotation rules), an LVLM (GPT-4o or Qwen2.5-VL-7B) outputs predicted categories {catⱼ} and per-category counts {numⱼ}. 2. Class-agnostic Mask Proposer: SAM2 (hiera-large) generates dense mask proposals {maskᵢ} via a regular point grid; image crops are also processed to improve small-object recall. 3. Counting-Constrained Mask-Label Matching: A semantic similarity matrix S ∈ ℝᴺˣᴹ is computed via GeoRSCLIP (ViT-L), where sᵢⱼ = cosine similarity between the CLIP image embedding of a 1.2× scaled crop around maskᵢ and the text embedding of catⱼ (template: "a satellite image of a {category}"). The assignment is solved as a Binary Integer Programming (BIP) problem minimizing total (1 − sᵢⱼ)·xᵢⱼ subject to (a) each mask assigned to at most one category, (b) assigned count per category exactly matches numⱼ (or all masks assigned if proposals are insufficient). BIP is solved with PuLP in ~0.07 s.

Key Contributions

  • InstructCDS task suite: Extends instruction-oriented detection to counting and segmentation under open-vocabulary, open-ended, and open-subclass settings.
  • EarthInstruct benchmark: Built from NWPU-VHR-10 and DIOR (20 categories, varying resolutions and annotation conventions); uses confidence-free metrics (mF1, mAPnc) and semantic similarity matching for open-ended evaluation.
  • InstructSAM framework: Training-free, confidence-free pipeline that integrates LVLM + SAM2 + domain CLIP via BIP matching; inference time is near-constant w.r.t. object count (R² = 0.01 vs. R² ≥ 0.94 for baselines).
  • Efficiency: 89% fewer output tokens and 32% lower total runtime versus direct bounding-box generation with Qwen2.5-VL in open-ended setting.

Results

  • Open-vocabulary (EarthInstruct, NWPU-VHR-10): InstructSAM-GPT4o achieves Cnt F1 = 83.0, Box F1 = 41.8, Mask F1 = 26.1 vs. Qwen2.5-VL (68.0 / 36.4 / —) and Grounding DINO (14.9 / 14.0 / —).
  • Open-vocabulary (DIOR): InstructSAM-GPT4o achieves Box F1 = 29.1, Mask F1 = 28.1 vs. Qwen2.5-VL (27.8 / —) and OWLv2 (14.3 / 1.2).
  • Open-ended (NWPU-VHR-10): InstructSAM-GPT4o Cnt F1 = 57.4, Box F1 = 31.3, Mask F1 = 29.9, outperforming LAE-Label (46.2 / 27.3 / —) and GeoPixel (40.8 / 29.9 / 29.1).
  • Open-ended (DIOR): InstructSAM-GPT4o Cnt F1 = 47.9, Box F1 = 22.1, Mask F1 = 21.8.
  • Ablation: Even InstructSAM-Qwen with smaller SAM2-S + SkyCLIP-B achieves Box F1 = 40.6, beating Qwen2.5-VL direct detection (36.4); performance scales with larger LVLM, SAM2, and RS-finetuned CLIP.
  • Inference: InstructSAM BIP solve = 0.07 s; mask proposal dominates runtime; runtime is effectively independent of object count.

Limitations

  • SAM2 misses objects with intricate geometry (e.g., basketball courts), propagating segmentation errors downstream.
  • GeoRSCLIP is trained on scene-level image–text pairs and performs suboptimally for object-level crop alignment, causing background confusion errors.
  • GPT-4o and SAM2 are trained primarily on optical imagery and fail on synthetic aperture radar (SAR) images.
  • Performance is bounded by and inherits biases from the constituent foundation models; no mechanism for end-to-end fine-tuning within the framework.

Relevance to Vision-Language Models

InstructSAM exemplifies a modular, training-free composition of VLMs (LVLM for instruction parsing + CLIP for semantic alignment) with a specialized segmentation foundation model (SAM2), demonstrating that structured prompt engineering and constrained combinatorial optimization can substitute for task-specific fine-tuning in a complex, domain-shifted setting. The framework highlights both the power and limits of zero-shot VLM transfer: strong LVLMs (GPT-4o) provide accurate object counts that act as global constraints, but domain-specific CLIP variants are still necessary for object-level visual grounding in aerial imagery. The confidence-free BIP formulation is directly relevant to VLM researchers developing evaluation protocols and inference strategies for generative detection models lacking calibrated confidence scores. EarthInstruct and the InstructCDS task suite extend the VLM benchmarking landscape to earth observation, a domain underrepresented in standard multimodal evaluation suites.