Skip to content

Object-Centric Refinement for Enhanced Zero-Shot Segmentation

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; object-centric region refinement enhances CLIP zero-shot segmentation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CLIP patch representations lack object-centric structure, which limits zero-shot segmentation of unseen categories. OC-ZSS injects frozen self-supervision-guided object prompts—whose attention is shaped by DINO clustering—into a frozen CLIP encoder, then applies a dual-stage Object Refinement Attention (ORA) module with multi-scale granular attention to iteratively refine both object and patch features. The result is state-of-the-art performance across inductive, transductive, and cross-domain zero-shot segmentation benchmarks.

Problem

Prior zero-shot segmentation (ZSS) methods built on frozen CLIP backbones focus exclusively on decoder-side improvements, ignoring that CLIP's patch-level features lack object-centric grouping. Without object-centric structure, patches cannot be naturally clustered into semantically coherent regions, which impairs fine-grained localization—especially for unseen categories where no labeled masks exist. Fixed-grid region prompts (e.g., CLIP-RC) fail to capture the non-rigid, irregular shapes of real objects, and methods that distill DINO (e.g., CLIP-DINOiser, ProxyCLIP) do not directly refine CLIP patch features.

Method

OC-ZSS operates on a frozen CLIP ViT-B/16 backbone with three added components:

Self-supervision-guided object prompts. A frozen DINO-B/16 encoder generates patch-level features that are clustered into \(n_o\) groups via Voronoi clustering. These cluster assignments define a custom attention mask \(\text{Attn\_mask} \in \mathbb{R}^{(1+n_p+n_o+M)\times(1+n_p+n_o+M)}\) (initialized to \(-\infty\) for object-prompt rows/columns, then zeroed for each prompt's assigned patch subset). Frozen object prompts \(O = \{O^1, \ldots, O^{n_o}\}\) are injected at every CLIP encoder layer under this masked attention, forcing each prompt to attend only to its DINO-assigned region and accumulate coarse object features \(OF = O^L\).

Dual-stage Object Refinement Attention (ORA). Run for \(S\) iterations. Each iteration has: 1. Object refinement: object features \(OF\) attend to patch features \(PF\) via cross-attention, then a GRU stabilizes the update: \(OF = \text{GRU}(OF_{\text{prev}},\, \text{CrossAttn}(OF, PF))\). 2. Patch refinement: updated \(OF\) serves as keys/values for patch cross-attention, again GRU-stabilized: \(PF = \text{GRU}(PF_{\text{prev}},\, \text{CrossAttn}(PF, OF))\).

Granular attention. Linear projections for keys/values in object refinement and queries in patch refinement are replaced by a multi-scale extractor using parallel depthwise separable atrous convolutions with four dilation rates plus global average pooling, concatenated and projected back to dimension \(D\): $\(X_{\text{gran}} = \text{Conv}_{1\times1}(\text{Concat}(\text{DWConv}_{r=d_1}, \ldots, \text{DWConv}_{r=d_4}, \text{GAP})(PF))\)$

The final refined \(PF\) replaces encoder outputs as keys/values in the 3-layer MPSA decoder. Loss is \(\mathcal{L}_\text{tot} = \mathcal{L}_\text{seg}(Y, Y_{gt}) + \mathcal{L}_\text{seg}(\tilde{Y}, Y_{gt})\) combining Dice and focal losses.

Key Contributions

  • Object prompts with masked attention: frozen prompts injected into CLIP layers that attend to DINO-clustered regions, extracting coarse object features without encoder fine-tuning or labeled masks.
  • Dual-stage ORA module: iterative cross-attention between object and patch features (with GRU stabilization) that mutually refines both representations and enriches patch-text alignment.
  • Granular attention: multi-scale depthwise separable atrous convolutions inside ORA that make refinement robust to objects of varying spatial scales.
  • Parameter efficiency: keeps CLIP and DINO entirely frozen; only visual prompts (VPT), ORA, and the lightweight MPSA decoder are trained.
  • State-of-the-art across three settings: inductive, transductive, and cross-domain ZSS on VOC 2012, PASCAL Context, and COCO-Stuff 164K.

Results

  • PASCAL VOC 2012 (inductive): 85.3% mIoU unseen (U), 93.6% seen (S), 89.2% hIoU — outperforms OTSeg+ and CLIP-RC.
  • PASCAL Context (inductive): 62.1% (U), 55.6% (S) — higher than prior methods.
  • COCO-Stuff 164K (inductive): 42.8% (U), 42.2% (S) — higher than prior methods.
  • State-of-the-art also reported for transductive and cross-domain settings (full tables truncated in provided text).
  • Baselines compared: ZegFormer, ZSSeg, MAFT, MaskCLIP+, ZegCLIP, CLIP-RC, Cascade-CLIP, OTSeg+, MVP-SEG+, SPT-SEG, AlignZeg.

Limitations

  • DINO clustering for attention masks is unsupervised and produces coarse, imperfect object groupings; downstream object features inherit this coarseness and rely on ORA to compensate.
  • Number of object prompts (\(n_o\)) is a dataset-level hyperparameter (6 for VOC/COCO, 8 for Context), requiring manual tuning per benchmark.
  • Voronoi clustering is a fixed, non-learnable assignment; errors in object-region assignment cannot be corrected by training.
  • Evaluation is confined to standard ZSS inductive/transductive splits; performance under fully open-vocabulary protocols is not assessed.
  • Computational overhead of DINO inference at test time (for attention mask generation) is not explicitly quantified.

Relevance to Vision-Language Models

OC-ZSS directly addresses a structural weakness of CLIP as a dense-prediction backbone: its patch tokens are optimized for global image-text alignment and lack the spatial grouping needed for pixel-level tasks. The paper demonstrates that injecting object-centric structure into a frozen CLIP encoder—rather than altering the decoder or distilling another model—significantly improves generalization to unseen visual categories, which is a core challenge for VLMs deployed in open-world scenarios. The use of DINO as a structural prior (not a feature distillation target) is a methodologically clean way to bridge the gap between self-supervised localization and language-grounded recognition. This connects to a broader line of work on making CLIP patch features spatially coherent (CLIP-DINOiser, ProxyCLIP, MaskCLIP+) and has direct implications for grounding, open-vocabulary detection, and any VLM application requiring fine-grained region-language alignment.