Towards Text-Mask Consistency in Medical Image Segmentation¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; text-mask consistency constraints fix VLM multi-lesion segmentation failures

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Medical VLMs for segmentation routinely produce masks that contradict the accompanying clinical text (wrong lesion count, wrong laterality) because templated reports cause false negatives in contrastive learning and because existing fusion is unidirectionally vision-dominant. C2Seg introduces soft-label cluster-aware contrastive pretraining and a bidirectional complementary attention module with KAN-based gating, achieving state-of-the-art segmentation on four public medical datasets while improving text–mask semantic consistency.

Problem¶

Two concrete failures in VLM-guided medical image segmentation: (1) clinical reports are highly templated (QaTa-COV19 has ~7,000 cases sharing only ~300 unique text templates), so standard InfoNCE hard-negative contrastive learning erroneously repels semantically identical text–image pairs, degrading cross-modal alignment; (2) existing "bidirectional" fusion methods (e.g., M3Att) update only text tokens in their language branch and still rely on a vision-dominant spatial representation, leaving textual semantics unable to impose pixel-level structural constraints such as lesion count, laterality, or coarse position.

Method¶

C2Seg is a two-stage framework:

Stage I — Cluster-aware Contrastive Learning (CaCL). Given a batch of \(B\) sentence embeddings \(\{l_i\}\), compute the text–text cosine similarity matrix \(M_{ij} = \cos(l_i, l_j)\), apply row-mean debiasing \(M'_{ij} = \max\{M_{ij} - \mu_i, 0\}\), then temperature-soften into targets \(\hat{Y}_{ij}\). The final target blends self-positive and neighbors: \(Y_{ij} = \rho\hat{Y}_{ij} + (1-\rho)\mathbf{1}[j=i]\). A symmetric bidirectional cross-entropy loss \(\mathcal{L}_\text{CaCL}\) replaces InfoNCE, so semantically near pairs receive nonzero target mass and their repulsive gradient is attenuated or reversed.

Stage II — Bidirectional Complementary Attention Module (BCAM). Given visual features \(V \in \mathbb{R}^{P \times C}\) and language features \(L \in \mathbb{R}^{N \times C}\), KAN layers produce keys/values for both modalities. A shared attention score matrix \(A = \frac{1}{\sqrt{d}} V_\text{key} L_\text{key}^\top \in \mathbb{R}^{P \times N}\) feeds two parallel paths: a vision-dominant path \(F_v = \text{softmax}_N(A) \cdot L_\text{value} \in \mathbb{R}^{P \times C}\) (each pixel aggregates token semantics), and a language-dominant path \(F_l = \frac{1}{N}\sum_n \text{softmax}_P(A^\top[n,:]) \odot V_\text{value} \in \mathbb{R}^{P \times C}\) (each token's spatial attention redistributed onto the pixel grid), preserving full spatial resolution for decoding.

KAN-based Attention Gating (K-Gate). Two independent two-layer KAN heads with tanh produce gating tensors \(g_v, g_l \in [-1,1]^{P \times C}\), element-wise reweight \(F_v\) and \(F_l\), then a \(1\times1\) convolution fuses the concatenated gated streams into \(F_\text{out}\). KAN layers also replace MLP blocks in the visual encoder and BCAM keys/values.

The language encoder (CLIP) is frozen throughout; only the visual encoder is fine-tuned at a small learning rate.

Key Contributions¶

CaCL: converts intra-batch text–text cosine similarity into soft contrastive labels, directly suppressing false-negative gradients from templated clinical language
BCAM: the first bidirectional fusion design where the language-dominant path explicitly produces a spatially resolved pixel-grid feature map (rather than updated text tokens only), enabling true language-to-vision spatial injection
K-Gate: KAN-based nonlinear gating that performs per-modality selective suppression before cross-modal mixing, improving fine-grained spatial feature selection
Gradient analysis proving that soft target mass \(Y_{ij} > 0\) attenuates or reverses repulsive gradients for semantically similar pairs (Eq. 3)
CLIP text encoder (despite not being medically pretrained) outperforms BioBERT and PubMedBERT in this framework, attributed to its aligned visual–language embedding space

Results¶

All metrics below are from Tables 1–3:

QaTa-COV19: C2Seg Dice 85.25% / mIoU 76.97% / HD95 12.71 / ASSD 3.38 — vs. prior best ARSeg 84.09% / 72.64% / 19.90 / 5.24 and MedLangViT 84.27% / 75.93% / 14.51 / 3.97
MosMedData+: Dice 77.81% / mIoU 65.17% / HD95 15.02 / ASSD 3.76 — vs. prior best MedLangViT 75.95% / 63.17% / 18.29 / 4.12
CVC-ClinicDB: Dice 91.82% / mIoU 86.81% / HD95 6.53 / ASSD 2.23 — vs. prior best MMIUNet 89.96% / 84.14% / 11.55 / 4.02
Kvasir: Dice 91.92% / mIoU 85.27% / HD95 13.62 / ASSD 3.98 — vs. prior best LAVT 90.83% / 84.90% / 15.90 / 4.15
C2Seg uses only 18.92M parameters, substantially fewer than most text-guided baselines (LAVT 118.6M, RefSegformer 195M, MedLangViT 27.7M)
Ablation (Table 3): step-wise gains — adding BCAM over DualA: +0.91% Dice on MosMedData+; adding K-Gate: +0.53%; switching HardCL→CaCL: +0.49%; full model: 77.81% vs. baseline 73.61%

Limitations¶

Language encoder is frozen; the framework has not been validated with an end-to-end trained or medically fine-tuned text encoder
Evaluation restricted to four datasets (two chest CT, two colonoscopy polyp); generalization to other modalities (MRI, dermatology, pathology) is untested
Text annotations for non-lung datasets are synthetically constructed following prior work rather than genuine clinical reports, which may not reflect real-world templating severity
Soft-label construction requires \(O(B^2 C)\) batch-level computation; very large batch sizes may introduce overhead (though the paper deems it negligible relative to the backbone)
No analysis of failure cases where CaCL soft labels may assign high similarity to semantically distinct but lexically similar templates

Relevance to Vision-Language Models¶

C2Seg directly addresses a known failure mode of contrastive VLM pretraining — false negatives from near-duplicate text — with a principled soft-label remedy grounded in gradient analysis, offering a transferable technique for any domain with highly templated natural language (e.g., radiology, legal, e-commerce). The BCAM design challenges the prevailing assumption in VLM fusion architectures that language need only modulate vision unidirectionally; it demonstrates that constructing an explicit spatially-resolved, language-dominant feature map improves pixel-level semantic grounding, a principle relevant to grounded segmentation models like GLIP, LISA, and SAM variants. The result that frozen CLIP outperforms domain-specific encoders (BioBERT, PubMedBERT) reinforces the broader finding that large-scale vision–language aligned pretraining yields more useful embeddings for downstream fusion than unimodal text pretraining alone. This work is a concrete instantiation of how to improve text–mask semantic consistency — a metric increasingly important as VLMs are evaluated on fine-grained referring segmentation tasks.