Skip to content

Class Distribution-induced Attention Map for Open-vocabulary Semantic Segmentations

πŸ•’ Published (v1): 2025-01-01 Β· Source: ICLR Β· Venue: ICLR 2025 Β· link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

πŸ’¬ Ask ChatGPT✦ Ask Claude

TL;DR

CDAM is a training-free plug-in that improves open-vocabulary semantic segmentation by replacing CLIP's last-layer attention with a Jensen-Shannon divergence map built from inter-patch class distribution similarities. It exploits the observation that CLIP's patch-wise class distributions are more reliably correlated within the same object than its argmax predictions are correct. Applied to existing CLIP-based methods, it yields consistent mIoU gains across six benchmarks without any fine-tuning.

Problem

CLIP-based training-free segmentation methods (MaskCLIP, SCLIP, GEM, ClearCLIP) produce patch-wise noisy class predictions that cause poor object localization. Using identity matrices or self-self (query-query, key-key) attention to modify the last CLIP layer does not leverage semantic class information, leaving high false-negative and false-positive rates for target classes.

Method

Core observation: Even when per-patch argmax class predictions are wrong (e.g., 67% accuracy on VOC21 with SCLIP), the full class-distribution vectors of same-object patches are highly correlated (~79% pairwise ranking accuracy). CDAM exploits this.

CDAM construction: Given an image, compute the dense similarity map S ∈ R^(N-1)Γ—C (patch-to-text cosine similarities, softmaxed) from an existing CLIP-based method. The CDAM replaces the last-layer attention:

AttnCDAM = Softmax({1 βˆ’ D_JS(Sβ€–S^T)} / Ο„)

where D_JS is the Jensen-Shannon divergence between every pair of patch class distributions. Patches with similar distributions get high attention weights.

Multi-scale CDAM (AttnMS): CDAMs computed at multiple downsampling scales (M = {0.25, 0.37, …, 1.0}) are averaged for spatial consistency.

Augmented text prompts (ATP): 80 attribute/super-category text classes (from PACO, COCO-Stuff) are appended to the text encoder input only during CDAM construction, enriching the class distribution representation without affecting final inference class predictions.

Entropy-based background thresholding: Foreground patches have low-entropy distributions (confident) and background patches have high-entropy distributions. The threshold is set as Thr_ent-bg = Ξ± Β· Thr_default / H(S)_center, where H(S)_center is the mean of max and min patch entropies, adapting dynamically per image.

CDAM integrates into the last CLIP attention layer by reusing latent features from layer Lβˆ’1 as value features and substituting AttnMS for the standard attention map.

Key Contributions

  • CDAM: training-free attention refinement using JS divergence between patch class distributions, replacing noisy argmax-based localization.
  • Multi-scale CDAM (AttnMS) for improved spatial consistency via downsampling/upsampling aggregation.
  • Augmented text prompts (80 attribute/super-category labels) to enrich class distribution similarity signals.
  • Entropy-based background thresholding that dynamically adapts per image without dataset-specific tuning.
  • Demonstrated plug-in compatibility: CDAM boosts MaskCLIP, SCLIP, ClearCLIP, and GEM without architectural changes.

Results

Datasets with background class (mIoU %): - MaskCLIP β†’ MaskCLIP+CDAM: VOC21 33.1β†’55.9 (+22.8), Context60 23.3β†’30.5 (+7.2), COCO-Obj 24.8β†’34.3 (+9.5), Avg 27.1β†’40.2 (+13.1) - SCLIP β†’ SCLIP+CDAM: VOC21 50.5β†’59.0 (+8.5), Context60 25.8β†’30.4 (+4.5), COCO-Obj 31.3β†’34.5 (+3.0), Avg 35.9β†’41.3 (+5.4) - GEM β†’ GEM+CDAM: VOC21 52.1β†’58.7 (+6.6), Context60 28.1β†’30.6 (+2.5), COCO-Obj 33.8β†’35.2 (+1.4), Avg 38.0β†’41.5 (+3.5) - MaskCLIP+CDAM outperforms all weakly-supervised methods (except PACL) on all three background datasets.

Datasets without background class (mIoU %): - MaskCLIP β†’ +CDAM: COCO-Stuff 16.5β†’24.5 (+8.0), CityScapes 23.8β†’27.6 (+3.8), ADE20K 12.2β†’17.8 (+5.6), Avg 17.5β†’23.3 (+5.8) - GEM β†’ +CDAM: COCO-Stuff 23.7β†’24.8 (+1.1), CityScapes 21.2β†’23.7 (+1.5), ADE20K 15.7β†’17.2 (+1.5), Avg 20.2β†’21.9 (+1.7)

Efficiency: CDAM adds at most 34 ms inference overhead; ~200Γ— faster than CaR on COCO-Obj.

Limitations

  • Improvements are smaller when there is no background class to threshold (Table 2 vs. Table 1), suggesting entropy thresholding drives much of the gain on background-inclusive benchmarks.
  • Incompatible with CLS-token-based classifiers (e.g., CaR, CLIP-DIY) because CDAM relies on local patch tokens.
  • ATP introduces additional text encoder calls for 80 auxiliary labels at inference, adding latency (though still minimal overall).
  • Method is validated only on CLIP ViT/B-16 with OpenCLIP weights; generalization to larger CLIP variants or non-ViT architectures is not demonstrated.
  • Background thresholding hyperparameter Ξ± (set to 2.5) is fixed globally; while not dataset-specific, it may not be optimal in all domains.

Relevance to Vision-Language Models

CDAM directly interrogates a known weakness of CLIP for dense prediction: image-level pretraining produces globally aligned but locally noisy features. The key insightβ€”that the full class-distribution vector is a more reliable inter-patch similarity signal than the argmax predictionβ€”is a practically useful diagnostic for any VLM being adapted to localization tasks. The plug-in design is broadly applicable: any model that produces a patch-to-text similarity map can use CDAM without retraining, making it relevant to the growing ecosystem of CLIP-derived architectures for grounding, referring expression comprehension, and open-vocabulary detection. It also illustrates how information-theoretic tools (JS divergence, entropy thresholding) can substitute for learnable components in VLM adaptation.