WIMFRIS: WIndow Mamba Fusion and Parameter Efficient Tuning for Referring Image Segmentation¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Window Mamba fusion with parameter-efficient tuning for referring segmentation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

WIMFRIS identifies that existing Parameter-Efficient Tuning (PET) methods for Referring Image Segmentation (RIS) neglect the neck module responsible for aggregated intermediate fusion, creating a performance bottleneck. It introduces the Hierarchical Mamba Fusion (HMF) block—using a Window Mamba Fuser (WMF) to perform SSM-based cross-modal fusion on non-overlapping window-partitioned features—alongside a complementary PET strategy of Mamba Text Adapter (MTA), Multi-Scale Aligner (MSA), and learnable emphasis parameters. WIMFRIS achieves state-of-the-art mIoU across all three standard RIS benchmarks with only 1.3–2.2% backbone parameter updates.

Problem¶

PET-based RIS methods adapt frozen CLIP/DINOv2 backbones via layer-wise adapters for vision–language alignment, but treat the neck module as an afterthought. Existing necks redundantly repeat the same basic cross-attention alignment already performed by adapters; or are omitted entirely, causing a significant performance drop (e.g., removing the neck from ETRIS reduces RefCOCO val mIoU from 74.5 to 72.2). The critical step of aggregating independently tuned multi-scale features into a unified representation and performing a single, powerful intermediate cross-modal fusion is unaddressed.

Method¶

WIMFRIS combines two complementary contributions:

HMF neck block. Three multi-scale visual features $\{f_v^i\}_{i=1}^{3}$ from DINOv2 are concatenated and projected via $1\times1$ convolution into a unified map $f_v^c \in \mathbb{R}^{H\times W\times C}$. The WMF module partitions $f_v^c$ into $n_{win} = HW/M$ non-overlapping windows of size $M = w_h \times w_w$, then prepends a shared textual class token $f_t^{cls}$ to each window, forming compact sequences $x_j \in \mathbb{R}^{(M+1)\times C}$. Mamba SSM scans all windows in parallel: $$Y = \mathrm{SSM}_j(X),\quad X \in \mathbb{R}^{n_{win}\times(M+1)\times C}$$ This constrains sequence length to $M+1$, directly mitigating the exponential decay problem of SSMs where distant tokens are attenuated. Fused visual features and updated textual tokens are recovered via attentional gating and window reassembly.

PET strategy. - MTA (Mamba Text Adapter): SSM-based adapter inserted into CLIP text encoder layers to produce a globally-aware textual prior $f_t$, using down/up-projection, 1D convolution, and SSM with linear complexity. - MSA (Multi-Scale Aligner): inserted into DINOv2 layers; a RFMixer applies depth-wise strip convolutions $\mathrm{DWConv}_{c_k\times 1} \circ \mathrm{DWConv}_{1\times c_k}$ at three scales to capture multi-scale context, followed by cross-attention with $f_t$ as key/value. - Emphasis parameters: per-stage learnable scalar $\alpha = \sigma(p) \in (0,1)$ modulates the adapter contribution $f_v^i = x_v^i + \alpha \cdot x_v^{i,\text{fused}}$, initialized at $\alpha_0 = 0.2$.

Training uses a composite loss $\mathcal{L} = 0.5\mathcal{L}_{con} + 0.3\mathcal{L}_{dice} + 0.2\mathcal{L}_{align}$, where $\mathcal{L}_{con}$ is text-to-pixel contrastive loss, $\mathcal{L}_{dice}$ is mask overlap loss, and $\mathcal{L}_{align}$ is per-window binary cross-entropy on WMF class token outputs.

Key Contributions¶

Empirical and ablation evidence that the neck's intermediate fusion role is distinct from and critical beyond layer-wise PET alignment.
HMF block with WMF: non-overlapping window partitioning + text-token prepending to mitigate SSM exponential decay in multimodal sequences.
MTA: SSM-based text adapter providing a globally-coherent textual prior for modality fusion.
MSA with RFMixer: multi-receptive-field strip convolution fused with cross-attention for precise layer-wise vision–language alignment.
Learnable sigmoid-parameterized emphasis scalars for adaptive stage-wise weighting.
WIMFRIS-L sets new SOTA across RefCOCO, RefCOCO+, and G-Ref with only 1.3–2.2% trainable parameters.

Results¶

WIMFRIS-B (DINOv2-B/14): average mIoU 71.8 across RefCOCO/RefCOCO+/G-Ref — surpasses most PET methods and matches all full fine-tuning methods.
WIMFRIS-L (DINOv2-L/14): average mIoU 73.4 — new SOTA over all PET and full fine-tuning approaches including DETRIS-L (72.2), RISCLIP-L (74.9, full fine-tuning on a subset).
WIMFRIS-L* (mixed RefCOCO/RefCOCO+/G-Ref training): average mIoU 77.7, surpassing DETRIS-L* (77.0) and UNINEXT-L* (76.0).
Neck ablation (RefCOCO, CLIP-ViT-B/16): inserting HMF into ETRIS raises val mIoU 72.2→74.5 (+2.3); into DETRIS raises 74.3→75.8 (+1.5); WIMFRIS achieves 77.2 val.
WMF window-size ablation: no partitioning (full concatenation) yields 75.7/77.3/72.6 (val/testA/testB); $4\times4$ window is optimal; $2\times2$ degrades performance due to excessive spatial fragmentation.

Limitations¶

Window size is a sensitive hyperparameter: overly small windows ($2\times2$) hurt performance by excessive spatial fragmentation; optimal $4\times4$ may not generalize across image resolutions.
Dual-encoder dependency (CLIP text + DINOv2 vision) increases memory and inference cost versus single-backbone PET methods.
WMF SSM scans are strictly within windows; cross-window dependencies are mediated only through the shared text class token, potentially limiting long-range spatial reasoning for spatially distributed referents.
No inference latency or FLOPs comparison is provided in the available text; computational overhead of the HMF neck is unquantified.
Evaluation is restricted to standard static-image RIS benchmarks; generalization to video RIS or open-set referring is not assessed.

Relevance to Vision-Language Models¶

WIMFRIS directly addresses how frozen VLM backbones (CLIP, DINOv2) should be efficiently adapted for fine-grained dense prediction, revealing that the aggregation and intermediate fusion stage is as important as layer-wise alignment—a design lesson broadly applicable to any VLM-based dense task (grounding, segmentation, detection). The use of Mamba SSMs as the fusion operator in a windowed, text-prepended scheme offers a computationally linear alternative to cross-attention in the neck, with an explicit mechanism to counter SSM's exponential locality bias in long multimodal sequences. This connects to the growing line of work exploring SSM-based VLMs and efficient multimodal fusion beyond transformers, and demonstrates that PET applied to dual-encoder VLMs requires careful architectural treatment of the feature aggregation stage—not just adapter design within individual encoder layers.