Multimodal Fusion for Fine-Grained Classification of Breast Fibroadenoma and Phyllodes Tumors¶

🕒 Published (v1): 2026-07-02 12:30 UTC · Source: Arxiv · link

Why this paper was selected

Medical VLM fusion for breast tumor classification; fine-grained multimodal clinical decision support

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

FA and phyllodes tumors (PT) are nearly indistinguishable on B-mode ultrasound, making fine-grained preoperative subtype classification clinically critical yet underexplored. This work introduces the FAPT-M dataset (910 pathology-confirmed patients with ultrasound images, structured clinical variables, and diagnostic text) and a multimodal framework using FiLM-conditioned adaptive modulation, cross-modal Transformer fusion, and dual-path representation learning. The proposed method reaches 77.64% accuracy and 89.74% AUC on a 3-class FA / benign-PT / borderline-PT task, outperforming single-modal and VLM-based baselines.

Problem¶

Most deep-learning methods for breast ultrasound target coarse benign-vs-malignant classification and rely on images alone. Fine-grained discrimination between FA, benign PT, and borderline PT—three visually similar classes with different surgical consequences—is underexplored, and no large-scale pathology-confirmed multimodal dataset existed for this task. Clinical diagnosis naturally integrates imaging, structured patient data, and report text, yet prior models ignore this complementary information.

Method¶

Three modality-specific encoders are applied independently: DenseNet121 for ultrasound images, a pretrained language encoder (CLIP-style) for diagnostic text descriptions, and an MLP for structured clinical variables (age, tumor size). All three are projected into a shared \(d\)-dimensional embedding space via learnable linear transforms followed by layer normalization:

\[z_m = \mathrm{LN}(W_m E_m(\cdot)), \quad m \in \{\text{img, txt, clin}\}\]

Clinical-Guided Adaptive Modulation (FiLM-gated): The clinical embedding \(z_\text{clin}\) generates channel-wise scale \(\gamma\) and shift \(\beta\) parameters, applied to image and text embeddings via a gated residual:

\[z_\text{mod} = z + g(z_\text{clin}) \cdot (\gamma \odot z + \beta)\]

where \(g(\cdot) \in (0,1)\) is a learnable gate that controls modulation strength, preventing over-disruption of unimodal semantics in limited-data regimes.

Transformer Fusion Head: The three (post-modulation) tokens are concatenated into a sequence \(T = [z^\text{mod}_\text{img}, z^\text{mod}_\text{txt}, z_\text{clin}]\) and processed by a multi-layer Transformer encoder with symmetric multi-head self-attention, yielding interaction-enhanced representations \([z^f_\text{img}, z^f_\text{txt}, z^f_\text{clin}]\).

Dual-Path Fusion: The final classifier receives both raw unimodal embeddings and the Transformer-refined embeddings concatenated:

\[z_\text{fusion} = \mathrm{Concat}(z_\text{img}, z_\text{txt}, z_\text{clin},\; z^f_\text{img}, z^f_\text{txt}, z^f_\text{clin})\]

This preserves high-frequency discriminative cues that global attention can over-smooth.

Key Contributions¶

Construction of FAPT-M, a 910-patient pathology-confirmed multimodal dataset with ultrasound images, structured clinical attributes (age, tumor size, BI-RADS), and free-text diagnostic descriptions, for three-class FA / benign-PT / borderline-PT classification.
A clinically conditioned FiLM modulation mechanism that injects structured clinical priors into image and text feature transformations via gated affine reparameterization, rather than late concatenation.
A Transformer-based cross-modal fusion head with symmetric self-attention across modality tokens, avoiding imposed hierarchical modality relationships.
A dual-path representation that concatenates raw and interaction-enhanced embeddings to prevent over-smoothing of locally discriminative cues.
Patient-level five-fold cross-validation protocol ensuring no data leakage; multi-center acquisition across six ultrasound device models.

Results¶

Proposed method: Accuracy 77.64%, F1 73.38%, AUC 89.74% (patient-level 5-fold CV, 3-class FA/benign-PT/borderline-PT).
Outperforms representative CNN-based, Transformer-based, and vision–language-based baselines (specific baseline numbers are not included in the provided text excerpt).
Ablation studies confirm that three-modality fusion (image + text + clinical) outperforms any two-modality subset.
Multi-center validation confirms generalization across different ultrasound acquisition devices.
Class-balanced evaluations confirm robustness given the unequal class distribution (410 FA / 211 benign PT / 255 borderline PT).

Limitations¶

Single-center retrospective dataset; multi-center prospective validation is still pending.
Text and clinical data were extracted from electronic medical records of one institution, potentially reflecting site-specific reporting style.
Malignant PT is excluded from the fine-grained classification task, limiting clinical completeness for the full PT spectrum.
The paper excerpt does not report specific baseline accuracy numbers, making quantitative comparison unverifiable from the provided text.
Dataset size (910 patients) is moderate; generalization to rare borderline PT cases may be limited.

Relevance to Vision-Language Models¶

This work directly benchmarks CLIP-inspired text encoding and vision–language baselines against a purpose-built multimodal medical fusion framework, providing evidence that off-the-shelf VLM representations alone are insufficient for fine-grained medical subtype classification. The FiLM-conditioned modulation of visual tokens by structured clinical embeddings represents a lightweight alternative to full cross-attention VLMs, particularly relevant for low-data medical settings where large VLMs are hard to fine-tune. The FAPT-M dataset and dual-path fusion design are practically useful to VLM researchers adapting general-purpose vision-language pre-training to medical imaging, where domain-specific structured metadata must be integrated beyond free-text prompts.