Skip to content

DGSeg: Dynamic Gating of Semantic-Spatial Guided Predictions for Reasoning Segmentation

🕒 Published (v1): 2026-07-06 08:10 UTC · Source: Arxiv · link

Why this paper was selected

Dynamic gating for reasoning segmentation bridges language queries to pixel masks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DGSeg addresses noisy and ambiguous intermediate target cues in reasoning segmentation by routing MLLM-generated semantic (text description) and spatial (bounding box) cues through separate segmentation branches, then fusing their predictions with a learned pixel-wise dynamic gating module. On the challenging ReasonSeg benchmark, the 7B-scale model achieves 69.6% / 67.3% gIoU on val/test under zero-shot conditions, surpassing all reported baselines.

Problem

Existing reasoning segmentation pipelines compress MLLM reasoning into sparse geometric cues (points, boxes) or latent token embeddings that are fed jointly into a single segmentation process. This creates two compounding problems: (1) lossy compression from rich language reasoning into a single cue type introduces referential ambiguity; and (2) feeding all cues into a unified representation allows any erroneous cue to contaminate the entire prediction. Iterative MLLM refinement approaches address noise but at prohibitive training and inference cost.

Method

DGSeg adopts a two-branch architecture built on Qwen2.5-VL (MLLM) and SAM3 (segmentation backbone):

  1. Dual cue generation: Given image \(I\) and query \(T\), the MLLM produces a chain-of-thought trace \(c_{\text{CoT}}\), a concise textual description \(c_{\text{sem}}\), and a bounding-box localization \(c_{\text{spa}}\) via \((c_{\text{CoT}}, c_{\text{sem}}, c_{\text{spa}}) = F_{\text{reason}}(I, T)\).

  2. Separate segmentation branches: \(c_{\text{sem}}\) and \(c_{\text{spa}}\) are processed independently by SAM3 to produce mask logits \(\ell_{\text{sem}}, \ell_{\text{spa}} \in \mathbb{R}^{H \times W}\).

  3. Dynamic gating: A lightweight two-layer convolutional module takes concatenated pixel-decoder embeddings \(f_{\text{sem}}, f_{\text{spa}}\) and predicts a pixel-wise fusion weight \(W = \sigma(F_{\text{dg}}(\text{Concat}(f_{\text{sem}}, f_{\text{spa}}))) \in [0,1]^{h \times w}\). The final logit is \(\ell = W^{\uparrow} \odot \ell_{\text{sem}} + (1 - W^{\uparrow}) \odot \ell_{\text{spa}}\).

  4. Two-stage training:

  5. Stage 1: MLLM finetuned via GRPO with a composite reward: format correctness \(\mathcal{R}_{\text{format}}\), bounding-box IoU \(\mathcal{R}_{\text{spatial}}\), and segmentation IoU of the semantic branch \(\mathcal{R}_{\text{semantic}}\).
  6. Stage 2: MLLM and SAM3 frozen; only the gating module trained with segmentation loss \(\mathcal{L}_{\text{seg}}\), a soft branch-quality supervision loss \(\mathcal{L}_w\) (BCE against softmax-normalized per-branch IoU targets \(W^*\)), and an entropy penalty \(\mathcal{L}_{\text{ent}}\) to prevent degenerate averaging. Loss is reweighted toward pixels where branches disagree.

Key Contributions

  • A dual-branch reasoning segmentation pipeline that isolates semantic and spatial cues into independent processing streams to prevent noise propagation.
  • A lightweight dynamic gating module providing adaptive pixel-wise fusion, supervised by relative branch quality rather than only final mask loss.
  • A composite RL reward for Stage 1 that uses the segmentation model's own mask preference as a proxy for semantic cue correctness, avoiding expensive category-level annotation.
  • State-of-the-art zero-shot results on ReasonSeg (69.6% / 67.3% gIoU val/test at 7B scale) with fewer trainable parameters than prior RL-based methods.

Results

  • ReasonSeg (zero-shot, 7B): 69.6% gIoU val / 67.3% gIoU test — outperforms SAM-Veteran (+1.4% val, +4.7% test), SAM3 Agent (+4.2% val, +4.7% test), CoPRS, and Seg-Zero.
  • ReasonSeg (zero-shot, 3B): 66.0% gIoU val / 60.0% gIoU test — outperforms Seg-Zero by 3.4% / 3.9% and CoPRS by 4.7% / 2.2%.
  • RefCOCO / RefCOCO+ / RefCOCOg (7B avg.): 76.9% cIoU, matching SAM-Veteran and surpassing Seg-Zero (76.4%), Seg-R1 (73.7%), and SAM-R1 (75.7%).
  • Ablation — fusion strategies (3B, ReasonSeg val/test): dynamic gating (66.0/60.0%) > MoE (65.6/60.6%) > learned scalar (64.0/57.8%) > confidence-based (64.1/57.2%) > average fusion (63.8/58.2%) > no fusion baseline (60.8/55.7%); oracle fusion upper bound is 67.9/62.8%.

Limitations

  • Oracle fusion analysis reveals a ~1.9% gIoU gap on val remains above the dynamic gating result, indicating the gating module does not yet fully realize the upper bound.
  • Training requires two separate optimization stages, adding pipeline complexity compared to end-to-end approaches.
  • Stage 2 trains with batch size 1 on two H100 GPUs; scaling behavior and compute cost relative to iterative refinement baselines are not fully characterized.
  • Evaluation is restricted to single-object reasoning segmentation; multi-target or counting scenarios are not addressed.
  • The semantic reward uses the segmentation model's mask IoU as a proxy for semantic correctness, which ties reward quality to SAM3's own biases.

Relevance to Vision-Language Models

DGSeg directly advances the paradigm of coupling MLLMs with specialized perception models — a central design pattern in grounded VLM research. By demonstrating that separating semantic and spatial reasoning outputs into independent processing streams, rather than collapsing them into a single prompt representation, substantially reduces noise propagation, the work provides a transferable principle for any VLM pipeline that generates intermediate cues for downstream visual tasks. The RL finetuning with a segmentation-model-derived semantic reward also illustrates a novel form of process reward for fine-grained VLM output that bypasses costly category annotation. This connects to broader debates in the VLM community about how to elicit spatially precise outputs from language-dominant models without requiring dense supervision.