Skip to content

OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OpenWorldSAM extends SAM2 to open-vocabulary segmentation by injecting language understanding via a lightweight 4.5M-parameter adapter built on a frozen BEiT-3 encoder, without retraining either backbone. It achieves state-of-the-art zero-shot performance across semantic, instance, and panoptic segmentation through two key mechanisms: positional tie-breaker embeddings (multi-instance disambiguation) and cross-attention soft prompting (language-to-visual grounding).

Problem

SAM/SAM2 lacks semantic understanding—it responds to visual prompts (points, boxes) but cannot interpret text. Prior attempts to add language awareness either use cascaded detect-then-classify pipelines (losing end-to-end grounding), rely on heavyweight LLMs (7B+ parameters, e.g., LISA, GLaMM), or are limited to single-instance queries (EVF-SAM). No existing approach unifies category-level and sentence-level text prompting with multi-instance segmentation at low parameter cost.

Method

OpenWorldSAM wraps SAM2 with a frozen BEiT-3-Large multi-modal encoder and a 4.5M-parameter trainable language adapter:

  1. Multi-modal encoding: Image and text prompt are jointly processed by BEiT-3 (early cross-modal fusion across all 24 Transformer layers), producing a 1024-D CLS token p_lang that already loosely localizes the concept.
  2. Prompt projection: A two-layer MLP maps p_lang ∈ ℝ¹⁰²⁴ → u ∈ ℝ²⁵⁶ to match SAM2's prompt channel dimension.
  3. Positional tie-breaker: K=20 learnable 256-D vectors {t₁,...,tₖ} perturb u into K distinct queries qᵢ = u + tᵢ, enabling multi-instance segmentation from a single text query. Trained with Hungarian matching against all ground-truth instances of class T.
  4. Cross-attention soft prompting: A 3-layer Transformer with alternating self-attention (query diversity) and cross-attention over SAM2's 64×64 level-3 image features refines each qᵢ into spatially grounded queries q′ᵢ.
  5. Mask decoding: Refined queries replace SAM2's visual prompt tokens; SAM2's frozen mask decoder outputs K masks with confidence scores. Inference derives semantic/instance/panoptic results via merging, NMS, and confidence filtering.

All SAM2 and BEiT-3 weights are frozen; only the MLP, tie-breakers, and soft-prompting Transformer are trained on COCO-Stuff (104K images, 25 epochs, single A100).

Key Contributions

  • A unified prompt interface supporting both category-level (semantic/instance/panoptic) and sentence-level (referring expression) segmentation within a single parameter set.
  • Positional tie-breaker mechanism: learnable perturbation vectors that disentangle multi-instance queries without explicit instance-count supervision.
  • Cross-attention soft prompting Transformer that grounds language-aware queries in SAM2's hierarchical visual features.
  • Demonstration that freezing both foundation model backbones and training only a 4.5M-parameter adapter outperforms fine-tuning BEiT-3 (674M parameters), which degrades due to underfitting on sparse category labels.
  • An Oracle-Prompts evaluation protocol exposing unfair penalization of prompt-conditioned models under global-matching evaluation on large-vocabulary datasets.

Results

Zero-shot open-vocabulary segmentation (Table 1, vs. X-Decoder-L and APE-B): - ADE20K-150: 60.4 mIoU (+30.8 over X-Decoder's 29.6; X-Decoder uses global-matching) - ADE20K-857: 33.1 mIoU (+23.9 over X-Decoder's 9.2) - PASCAL Context-459: 47.5 mIoU (+25.7 over APE's 21.0; +31.4 over X-Decoder's 16.1) - ScanNet-40: 55.6 mIoU (+25.9 over X-Decoder's 29.7) - PASCAL VOC-20: 98.0 mIoU (comparable to prior SOTA ~98.3) - SUN-RGBD-37: 67.7 mIoU (+24.7 over X-Decoder's 43.0)

Oracle-Prompts evaluation (Table 2, fairest comparison): - ADE20K-857: 33.1 vs. X-Decoder Oracle 29.1 (+4.0) - PASCAL Context-459: 47.5 vs. X-Decoder Oracle 42.3 (+5.2) - ScanNet-40: 55.6 vs. X-Decoder Oracle 49.1 (+6.5) - PASCAL Context-59: 73.7 vs. X-Decoder Oracle 75.5 (−1.8; X-Decoder benefits from class memorization on common COCO-overlap categories)

Referring expression segmentation (Table 3, RefCOCOg UMD val): - 74.0 cIoU with 4.5M trainable parameters - Surpasses LISA (66.4), GLaMM (74.2 but with 40M params + much larger data), UNINEXT (74.4 but 673M params) - Below EVF-SAM (77.0), which is single-instance only

Instance segmentation (AP): underperforms APE (which uses Objects365/LVIS detection data); AP 16.9 vs. APE 29.0 on ADE-150.

Limitations

  • AP underperformance: Instance AP lags behind APE, which benefits from large detection-specific datasets (Objects365, LVIS) not used by OpenWorldSAM.
  • Single-text-query collapse without tie-breakers: Without the positional tie-breaker, the model segments only the most salient instance; tie-breakers require K to be set heuristically (K=20 for COCO).
  • Two-stage inference benefit is marginal: Quantitatively provides minimal improvement; only improves visual mask boundary quality.
  • Oracle-prompt dependency: The model's evaluation protocol requires ground-truth class names as prompts, which may not reflect realistic closed-set deployment scenarios.
  • Fine-grained category confusion: On smaller-vocabulary datasets where COCO-overlap categories are common (PC-59, VOC-20), X-Decoder under oracle prompts slightly outperforms, suggesting memorization effects advantage competing models.
  • Text-only BEiT-3 causes catastrophic collapse: The model cannot function with text prompts alone—visual co-encoding is mandatory, limiting use without image context.

Relevance to Vision-Language Models

OpenWorldSAM exemplifies a parameter-efficient VLM adaptation paradigm: rather than fine-tuning a large LLM for grounded segmentation, it extracts frozen cross-modal representations from BEiT-3 and bridges them to a frozen segmentation foundation model (SAM2) with a minimal adapter. The finding that early cross-modal fusion (BEiT-3) outperforms late-fusion CLIP by +33.9 mIoU is a direct and quantified argument about VLM architecture choice for pixel-level tasks. For VLM researchers, the paper also raises a methodological point: global-matching evaluation systematically underestimates prompt-conditioned models on large-vocabulary benchmarks, and the oracle-prompts protocol better isolates spatial grounding ability from vocabulary disambiguation. The tie-breaker mechanism is relevant to any VLM application requiring one-to-many grounding from a single semantic query.