Skip to content

MMR: A Large-scale Benchmark Dataset for Multi-target and Multi-granularity Reasoning Segmentation

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MMR is a 194K-pair dataset for reasoning segmentation that simultaneously addresses multi-target and part-level (multi-granularity) understanding, both absent in prior benchmarks. The authors also propose M²SA, a model extension of LISA that adds early local feature fusion and multiple [SEG] tokens to handle these two dimensions jointly. M²SA outperforms LISA, GSVA, and GLaMM on the new benchmark and on existing multi-target/multi-granularity RES tasks.

Problem

Existing reasoning segmentation datasets (ReasonSeg: 1.2K pairs, MUSE: 214K pairs) are limited: ReasonSeg targets single objects at object level; MUSE adds multi-target but stays at object level; neither supports implicit, hierarchically grounded part-level reasoning. This prevents VLMs from being used in robotics/interaction scenarios where understanding both which objects and which specific parts to act on is essential.

Method

Dataset construction (MMR): Images and masks are sourced from PACO-LVIS (75 object categories, 456 part classes, 273K object masks, 502K part masks). GPT-4V is prompted with a two-step pipeline: (1) generate a global caption, then (2) generate multi-round QA pairs requiring implicit/commonsense reasoning about object–part relationships, functions, and multi-target answers. Four human inspectors filter 12.6% of pairs using criteria of logicality, coherence, and clarity; a pair is removed only if flagged by ≥2 inspectors.

Model (M²SA): Built on LISA (LLaVA + SAM). Two modifications: 1. Early local feature fusion: features from SAM ViT-H layer 8 (fine-grained edges/boundaries) are passed through two conv layers and summed with global features from the final SAM encoder layer before the mask decoder. 2. Multiple [SEG] tokens: one [SEG] token is appended per target entity in the answer string; each token is independently projected into SAM's prompt space to produce a separate mask. Training uses cross-entropy (text) + weighted BCE+DICE (masks).

Key Contributions

  • MMR dataset: 194K QA pairs, 57,643 images, covering object-level, part-level, and mixed multi-target scenarios; train/val/test split of 154K/8K/32K.
  • Three test subsets: object-only, part-only, and mixed, enabling granularity-specific evaluation.
  • M²SA framework: early local feature fusion + multiple [SEG] tokens on top of LISA/LLaVA-7B and LLaVA-Llama2-13B backbones.
  • Demonstration that all prior models still have substantial headroom on part-level and multi-target segmentation.

Results

  • MMR benchmark (test, Obj & Part, cIoU): M²SA-7B: 46.8; GLaMMtr: 45.0; GSVA-7Btr: 34.5; LISA-7Btr: 27.1. M²SA-Llama2-13B: 47.6.
  • MMR part-only (test, cIoU): M²SA-7B: 27.0; GLaMMtr: 25.5; LISA-7Btr: 13.1. Pre-trained LISA-7B: 7.9 (shows severe weakness without MMR training).
  • Multi-RefCOCO (val, cIoU): M²SA-7B: 71.3 vs. GSVA-7B: 50.7 vs. LISA-7B: 34.0 — large margin.
  • RefCOCOm multi-granularity (val, mIoU, Obj & Part): M²SA-7B: 35.5 vs. UniRES: 34.3 vs. GLaMM: 35.3; M²SA-13B: 37.3 (best).
  • Single-target RefCOCO (val, cIoU): M²SA-7B: 74.6, competitive with but not exceeding LISA-7B (79.5), reflecting the single-target task is not its primary target.

Limitations

  • MMR excludes the "person" class (absent in PACO-LVIS object scope), causing weaker performance on RefCOCOm which is person-heavy.
  • Dataset images and masks are inherited from PACO-LVIS, so category diversity is bounded by that source (75 object, 445 part classes).
  • PixelLM excluded from baselines due to code reproducibility issues, leaving a gap in comparisons.
  • Average 1.8 targets per QA pair; while the max is 16, the distribution is skewed toward low target counts, so very high multi-target scenarios are underrepresented.
  • No evaluation of model robustness to paraphrased or adversarially perturbed implicit queries.

Relevance to Vision-Language Models

MMR directly benchmarks the ability of MLLMs (LLaVA, CLIP+LLM combos) to translate implicit natural language into pixel-level masks at both object and part granularity — a capability gap that standard VLM evaluations do not expose. The multiple [SEG] token mechanism is a lightweight, architecture-level technique for extending any LLM-based segmentation system to multi-output scenarios without architectural overhaul. For researchers tracking VLMs, this work identifies part-level grounding as an open frontier: even the strongest baselines score below 30 cIoU on part-only sets. The GPT-4V-assisted dataset generation pipeline is also a reusable template for scaling reasoning-grounded annotation to new domains.