Skip to content

RadZero: Similarity-Based Cross-Attention for Explainable Vision-Language Alignment in Chest X-ray with Zero-Shot Multi-Task Capability

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RadZero is a chest X-ray vision-language alignment framework that replaces softmax-based attention maps with cosine similarity-based cross-attention (VL-CABS) to produce calibrated, pixel-level image–text similarity maps. Combined with LLM-extracted finding-sentences and multi-positive contrastive training, it achieves zero-shot SOTA on classification, grounding, and segmentation benchmarks while providing principled explainability.

Problem

Existing medical VLMs suffer from two intertwined problems: (1) radiology reports are underutilized—prior methods align at word/token level or randomly sample one sentence per image, losing semantic richness; (2) softmax-based attention maps are poor explainability proxies—they always peak somewhere due to softmax normalization, and raw logits are unnormalized and inconsistently scaled across image–text pairs, making thresholding unreliable.

Method

Finding-sentence extraction: Llama-3.3-70B-Instruct decomposes each radiology report into minimal semantic units of the form "There is [finding] of [location]," yielding on average 6.45 sentences per MIMIC-CXR study. At inference, prompts are aligned with the "There is" prefix.

VL-CABS (Vision-Language Cross-Attention Based on Similarity): The frozen XrayDINOv2 encoder (DINOv2 fine-tuned on CXRs) produces patch embeddings; two trainable Transformer layers on top embed the high-resolution (518×518) input into L=1369 patch tokens. Text is encoded with fine-tuned MPNet (Sentence-BERT). All embeddings are ℓ2-normalized; scaled cosine similarity is computed between each patch and each text embedding, softmax over patches produces attention weights, and the attended vision embedding is re-normalized and dot-producted with the text embedding to give a global similarity logit. The patch similarity scores form a pixel-level VL similarity map after bilinear interpolation and sigmoid activation—directly interpretable as image–text similarity with a fixed [−1,1]-bounded, zero-centered scale.

Training objective: MP-NCE loss (UniCLIP) treats each of the Ni finding-sentences for image Ii as independent positives against all other sentences in the mini-batch, paired with a symmetric InfoNCE loss over images.

Zero-shot inference: Sigmoid of the global logit → classification probability; sigmoid of the resized patch-similarity map → grounding and segmentation; single threshold over the similarity map → open-vocabulary semantic segmentation.

Key Contributions

  • VL-CABS: cosine similarity-based cross-attention that produces calibrated, zero-centered VL similarity maps enabling consistent thresholding and interpretable pixel-level explanations.
  • LLM-driven finding-sentence extraction that decomposes reports into minimal semantic units for independent sentence-level embedding.
  • Multi-positive contrastive training (MP-NCE) to exploit all Ni finding-sentences per image at every training step rather than random single-sentence sampling.
  • High-resolution (518×518) support via positional embedding interpolation on a frozen vision encoder plus lightweight trainable Transformer layers.
  • Zero-shot multi-task capability (classification, grounding, phrase grounding, segmentation, open-vocabulary segmentation) from a single model without task-specific heads or mask annotations.

Results

  • Classification (AUROC): SOTA on Open-I (0.851 at 224px vs. CARZero 0.838) and PadChest (0.841 vs. 0.810); outperforms CARZero by 3.1 pp on PadChest (192 classes) and 4.2 pp on PadChest20 (rare diseases); average AUC across all datasets exceeds CARZero by 1.5 pp.
  • Grounding (pointing game, CXD10): RadZero mean 0.622 vs. CARZero 0.543 (+0.079); best per-class in 9/10 diseases.
  • Phrase grounding (MS-CXR): 0.844 vs. CARZero 0.749 (+9.5 pp), BioViL-T 0.719.
  • Segmentation Dice: SIIM 0.238 vs. CARZero 0.121 (+71% relative); RSNA 0.546 vs. CARZero 0.540; outperforms MGCA (1% supervised) on both datasets.
  • Pixel-AUC (SIIM): RadZero 0.976 vs. CARZero 0.856, MedKLIP 0.648, even vs. MGCA (10%) 0.943.
  • Ablation: Switching dot-product → cosine similarity alone improves grounding; Transformer vs. linear layers adds further gains; MP-NCE on top improves PC20 0.860→0.879 and SIIM 0.099→0.121; 518px vs. 224px boosts grounding 0.537→0.622 and SIIM 0.121→0.171.

Limitations

  • Performance degradation on CXR14, CXP, and CXD10 classification relative to CARZero; RadZero is not uniformly best across all datasets.
  • Pre-trained vision encoder (XrayDINOv2) was trained at 224px, causing the 224px model to sometimes outperform 518px on classification—an encoder pre-trained at higher resolution could close this gap.
  • Validated only on chest X-ray; generalization to CT, MRI, or natural image domains is unverified.
  • Segmentation masks for rare findings may extend beyond ground-truth bounding boxes; open-vocabulary segmentation boundaries remain imprecise.
  • Relies on a large proprietary-scale LLM (Llama-3.3-70B) for offline finding-sentence extraction, adding infrastructure cost.

Relevance to Vision-Language Models

RadZero directly advances the alignment interpretability problem central to medical VLMs by replacing the de facto standard (softmax attention maps) with geometrically well-defined cosine similarity maps, providing a principled pathway toward spatially grounded, auditable VL reasoning. The VL-CABS mechanism is a generalization of patch-level alignment (pioneered by GLoRIA, extended by dino.txt) that harmonizes training-time contrastive objectives with inference-time similarity computation—a consistency typically broken in cross-attention decoders. For the broader VLM community, the multi-positive NCE formulation and LLM-assisted text decomposition are directly transferable to any domain where a single image corresponds to multiple, semantically distinct textual descriptions. The zero-shot open-vocabulary segmentation capability—achieved without masks and with a single global threshold—suggests VL-CABS could be a plug-in module for making dense-prediction VLMs more interpretable beyond radiology.