Skip to content

FineRS: Fine-grained Reasoning and Segmentation of Small Objects with Reinforcement Learning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FineRS is a two-stage MLLM framework that unifies instruction-guided reasoning and pixel-level segmentation of ultra-small objects in high-resolution (4K) images using reinforcement learning. A Global Semantic Exploration (GSE) stage produces a coarse region and text answer, then a Localized Perceptual Refinement (LPR) stage generates a precise bounding box and SAM2 mask. A novel locate-informed retrospective reward trains GSE using LPR's localization quality as feedback, enabling data-efficient multi-task optimization without heavy supervised fine-tuning.

Problem

Existing MLLM-based reasoning and segmentation methods (e.g., LISA, SegZero) are designed for standard-resolution images with large, salient objects. When applied to high-resolution scenes containing extra-small objects (e.g., UAV imagery), they suffer from severe detail loss due to downsampling and cannot simultaneously produce accurate text answers and segmentation masks. Training-free high-resolution methods (DC2, MLLMs-Know) avoid this via image patching but lack precise spatial grounding, while vision-RFT methods (SegZero) support only fixed reward paradigms incompatible with diverse VQA formats.

Method

FineRS adopts a coarse-to-fine two-stage pipeline, both stages built on Qwen2.5-VL-7B, trained with GRPO-based reinforcement learning:

  1. GSE (Global Semantic Exploration): Takes full 1920×1080 HR image and user instruction; outputs a text response and a coarse bounding region (fixed-size box, optimizing only center offset) containing the referred object.
  2. LPR (Localized Perceptual Refinement): Crops the coarse region (upsampled to 512×512) and generates a refined bounding box and two prompt points, which are fed to a frozen SAM2 to produce the final segmentation mask.
  3. Training order: LPR is trained first with spatial rewards (Box L1, Box IoU, Point L1, JSON format, Q&A accuracy, think-format). GSE is then trained using a locate-informed retrospective reward: for each sample, n randomly offset coarse regions are generated, LPR selects the one yielding the highest box IoU with ground truth, and that region becomes the pseudo-GT for GSE's region IoU/L1 rewards. Additional GSE rewards enforce region size compatibility and full coverage of the target (box-in-region). No joint end-to-end training due to memory constraints.

Key Contributions

  • FineRS framework: First method to unify instruction-guided reasoning and fine-grained segmentation under a reinforcement learning paradigm using a coarse-to-fine two-stage MLLM.
  • Locate-informed retrospective reward: Cross-stage RL signal where LPR's localization quality retroactively supervises GSE's coarse region generation, without requiring explicit GT coarse regions.
  • Multi-task reward pool: Supports IS, MVQA, and OVQA within a single reward framework (response reward defined per-task type: keyword match for IS, exact option match for MVQA, fuzzy-match ≥0.8 for OVQA).
  • FineRS-4k dataset: First UAV-captured 4K-resolution benchmark with triplet annotations (question, answer, mask) across IS/MVQA/OVQA for ultra-small objects (S/XS/XXS size tiers); 12,132 text-mask pairs over 4,563 images.

Results

On FineRS-4k test set (all object sizes, all tasks): - FineRS 7B: gIoU 55.1 / cIoU 46.5, MVQA 83.3%, OVQA 56.7% - Best retrained baseline (Seg-Zero† 7B): gIoU 46.6 / cIoU 38.6 (no QA reported) - MLLMs-Know 13B + LISA 13B (training-free): gIoU 17.9 / cIoU 12.8, MVQA 52.6%, OVQA 48.8%

On V* benchmark (general HR, no finetuning): FineRS 77.5% overall vs. SEAL 75.4%, DC2 57.3%

On HR-Bench 4K / 8K (no finetuning): - HR-Bench 4K avg: FineRS 63.8% vs. DC2 50.0%, MLLMs-Know 41.3% - HR-Bench 8K avg: FineRS 58.1% vs. DC2 40.8%, MLLMs-Know 38.9%

Ablation (FineRS-4k test): - Removing retrospective reward: gIoU −1.1, cIoU −2.5, MVQA −1.0 - Removing box-size/box-in-region rewards: gIoU −4.1/−5.0, MVQA −26.8/−26.8 (largest drop)

Limitations

  • LPR localization fails when the target object falls outside GSE's coarse region, causing cascading errors with no recovery mechanism.
  • The two stages are trained sequentially and independently; joint end-to-end optimization is not performed due to GPU memory constraints, potentially leaving cross-stage synergy on the table.
  • FineRS-4k is restricted to UAV/drone imagery; generalization to other small-object domains (e.g., medical, satellite) is not evaluated.
  • SegZero† and other baselines retrained on FineRS-4k do not report QA metrics, making direct multi-task comparison incomplete.

Relevance to Vision-Language Models

FineRS demonstrates that vision RFT (GRPO-based RL) can be extended beyond single-task detection/segmentation to unify grounded reasoning and pixel-level localization in a single MLLM framework — a capability gap that persists even in recent models like SegZero. The locate-informed retrospective reward is a generalizable cross-stage RL coupling technique relevant to any multi-stage VLM pipeline where intermediate pseudo-labels are unavailable. The FineRS-4k benchmark fills a critical hole in VLM evaluation by targeting UAV-scale ultra-small objects with joint VQA+segmentation annotations, pushing resolution and object-size extremes well beyond standard benchmarks like ReasonSeg and refCOCOg. For researchers tracking VLMs, this work advances the frontier of grounded multimodal reasoning under low-data, high-resolution conditions.