Skip to content

Region-Grounded Vision-Language Learning for Detection-Guided Mammographic Lesion Classification

🕒 Published (v1): 2026-07-17 04:27 UTC · Source: Arxiv · link

Why this paper was selected

Region-grounded contrastive VL for mammography; spatial alignment beats global CLIP

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Global image–text contrastive alignment (à la CLIP) is ill-suited for mammography because lesions occupy less than 2% of the image area, causing diagnostically critical morphological cues to be diluted. This paper introduces a two-stage region-grounded VLM framework that aligns lesion-ROI embeddings with structured clinical text descriptors, jointly trained with an auxiliary FCOS-style detection head. The method outperforms prior VLMs and detection baselines on CBIS-DDSM and VinDr-Mammo across in-domain, zero-shot, and transfer learning settings.

Problem

Standard CLIP-style global image–text alignment fails in mammographic lesion classification due to three compounding issues: (1) lesion dilution — lesions occupy \(<2\%\) of full-field mammogram area, so global embeddings capture background parenchyma rather than lesion morphology; (2) semantic collapse — mammographic metadata has a small, structured vocabulary, causing contrastive learning to collapse onto generic lesion-presence signals rather than fine-grained morphological attributes (shape, margin, spiculation); (3) background bias — normal breast tissue dominates the image and pulls text descriptors toward irrelevant regions.

Method

Stage 1 — Region–Text Contrastive Pretraining. Visual features are extracted from a MammoCLIP-initialized backbone; ROIAlign is applied to annotated lesion boxes to yield region embeddings \(v_i\). Text embeddings \(t_i\) are constructed from a fixed template ("breast type {mass/calcification} with shape {shape} and margin {margin}"). The training objective is:

\[\mathcal{L}_{\text{stage1}} = \mathcal{L}_{\text{pos}} + \lambda_{\text{sem}} \mathcal{L}_{\text{sem-neg}} + \lambda_{\text{bg}} \mathcal{L}_{\text{bg-neg}}\]
  • \(\mathcal{L}_{\text{pos}}\): symmetric InfoNCE (CLIP-style) between region and text embeddings; duplicate structured captions in a batch are treated as multiple positives.
  • \(\mathcal{L}_{\text{sem-neg}}\): margin-ranking loss using attribute-swapped hard negatives (e.g., swapping shape or margin values), enforcing \(\text{sim}(v_i, t^+_i) - \text{sim}(v_i, t^-_{i,k}) > m\).
  • \(\mathcal{L}_{\text{bg-neg}}\): penalizes high similarity between background-sampled ROIs (low IoU to lesion annotations) and the paired lesion text embedding, with threshold \(\delta\).

Stage 2 — Detection-Guided Joint Training. A lightweight anchor-free FCOS-style detection head is appended to the shared encoder. The total loss is \(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{contrastive}} + \mathcal{L}_{\text{det}}\), where \(\mathcal{L}_{\text{det}}\) combines focal loss for objectness and smooth-\(L_1\) for bounding box regression. At inference, classification uses prompt-based scoring: ROI embeddings from detected lesions are compared against "benign lesion" / "malignant lesion" text prompts. A confidence-gated insurance mechanism falls back to the global image embedding when detection confidence falls below threshold \(\tau = 0.2\), triggered in 3–8% of test cases.

Key Contributions

  • Region-level contrastive pretraining via ROIAlign applied to annotated lesion boxes, replacing global image–text alignment.
  • Multi-component objective combining positive InfoNCE, attribute-swapped semantic hard negatives, and background suppression to prevent semantic collapse and background bias in low-vocabulary settings.
  • Joint auxiliary FCOS-style detection head that preserves spatial sensitivity and enables localization-aware classification through detection-guided ROI proposals.
  • Confidence-gated insurance that gracefully degrades to global embedding when lesion detection confidence is low.

Results

Classification (CBIS-DDSM, in-domain, mass-only): - Proposed: Acc 80.5 / F1 75.2 / AUC 85.5 vs. MammoCLIP: 77.4 / 74.6 / 83.7; FVLM: 76.7 / 74.4 / 84.6.

Classification (zero-shot cross-dataset, DDSM→VinDr, mass-only): - Proposed: Acc 61.2 / F1 67.6 / AUC 68.5 vs. MammoCLIP: 53.6 / 58.3 / 56.1; FVLM: 60.7 / 58.7 / 63.3.

Classification (transfer learning, DDSM+VinDr→VinDr, mass-only): - Proposed: Acc 73.1 / F1 70.4 / AUC 77.8 vs. MammoCLIP: 66.3 / 69.0 / 74.0.

Detection (mAP50, in-domain DDSM, mass-only): - Proposed: 58.1 vs. MammoCLIP: 57.4; YOLOv12: 56.6; YOLOv5: 49.8.

Detection (mAP50, zero-shot DDSM→VinDr, mass+calc): - Proposed: 27.2 vs. MammoCLIP: 14.1; YOLOv12: 9.2 — largest margin.

Ablation: Removing the detection head causes the largest classification drop; removing Stage 1 pretraining entirely compounds all degradations; removing background suppression or hard negatives each produce measurable declines in both tasks.

High-resolution (1520Ă—912): Proposed still outperforms MammoCLIP at the same resolution (e.g., mass AUC 85.7 vs. 84.1 on DDSM).

Limitations

  • Datasets are relatively small (CBIS-DDSM: ~1,600 mass cases; VinDr-Mammo: ~1,200 mass cases), limiting generalizability claims.
  • Clinical text descriptors are constructed from a fixed, low-vocabulary template; richer radiology reports are not leveraged.
  • The method requires bounding-box lesion annotations for ROIAlign during training, which constrains applicability to weakly or unlabeled data.
  • Confidence-gated insurance falls back to global embeddings (an acknowledged imperfect fallback) for 3–8% of test cases, introducing inconsistency.
  • Evaluation is limited to binary benign/malignant classification; BI-RADS grading or multi-class settings are not explored.
  • Extension to other modalities or longitudinal data is stated as future work, not demonstrated.

Relevance to Vision-Language Models

This work directly addresses a structural limitation of CLIP-style VLMs when applied to small-lesion medical imaging: global image–text alignment loses spatially localized diagnostic signal, a problem likely shared by any domain where targets are a tiny fraction of the image. The proposed region-grounded pretraining with ROIAlign and structured hard negatives is a concrete recipe for adapting general VLM training objectives to low-vocabulary, high-precision clinical settings. For VLM researchers, the multi-component loss (positive InfoNCE + attribute-swap hard negatives + background suppression) offers a transferable design pattern for preventing semantic collapse in structured-caption domains. The detection-classification joint training paradigm also extends the emerging line of grounded VLMs (RegionCLIP, GLIP) into a challenging real-world low-data medical domain.