Brain-Adapter: A Dual-Stream Vision-Language MIL Framework for Comprehensive 3D CT Diagnosis of Acute Intracranial Pathologies¶

🕒 Published (v1): 2026-06-22 15:41 UTC · Source: Arxiv · Venue: MICCAI 2026 · link

Why this paper was selected

MICCAI 2026; dual-stream MIL VLM for 3D CT; directly applicable to medical foundation model work

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Brain-Adapter adapts frozen 2D biomedical VLMs (BiomedCLIP) to 3D head CT multi-label classification via a dual-stream MIL framework supervised entirely by raw diagnostic reports—no dense pixel annotations required. A Text-Conditioned Attention (TCA) branch aligns slice features with fine-grained pathology descriptions, while a parallel ABMIL branch learns global morphological patterns from LLM-extracted structured labels. At inference, an Uncertainty-Aware Refinement module dynamically fuses both streams, achieving state-of-the-art performance on a 21-class acute intracranial pathology dataset.

Problem¶

Native 3D VLMs are scarce due to the lack of paired 3D-text corpora, and direct 3D deep learning models require expensive dense annotations while lacking semantic understanding. Standard 2D MIL approaches applied to volumetric CT operate purely in the visual domain, struggling to localize subtle pathologies without semantic guidance about which diagnostic concepts to search for.

Method¶

Brain-Adapter treats a thick-slice CT scan $X = \{x_i\}_{i=1}^N$ as a MIL bag of 2D slices and adapts frozen BiomedCLIP encoders via LoRA (rank $r=16$) on the vision attention layers only.

LLM-driven preprocessing: Raw radiology reports are parsed by an LLM into a structured multi-label set $Y \in \{0,1\}^C$ ($C=21$ classes across hemorrhagic findings, other pathologies, and etiology context).

Open-set Alignment branch (TCA): For each fine-grained pathological description $k$, its text embedding $t_k$ serves as a Query against slice feature Keys/Values: $$A_k = \text{Softmax}\!\left(\frac{t_k V^\top}{\sqrt{d}}\right), \quad h_k = A_k V$$ An InfoNCE loss aligns $h_k$ with $t_k$ while repelling other pathology embeddings: $$\mathcal{L}_{\text{align}} = -\sum_{k=1}^K \log \frac{\exp(\cos(h_k,t_k)/\tau)}{\sum_{j=1}^K \exp(\cos(h_k,t_j)/\tau)}$$

Logic-set Supervision branch (ABMIL): Gated attention-based MIL aggregates a global scan representation $z = \sum_i a_i v_i$, supervised by the LLM-extracted labels via Asymmetric Loss (ASL) to handle severe class imbalance.

Consistency constraint: Cosine distance between $z$ and $h_0$ (the TCA output for the global report embedding) enforces cross-stream coherence: $\mathcal{L}_{\text{cons}} = 1 - z^\top h_0 / (\|z\|\|h_0\|)$.

Uncertainty-Aware Refinement (UAR): At inference, per-class uncertainty $u_c = 1 - |2P_c - 1|^\alpha$ gates how much the TCA-derived semantic score $S_{\text{text}}^{(c)} = \sigma(\cos(h_c, t_c)/\tau)$ overrides the MIL prediction: $$P_{\text{refined}}^{(c)} = (1 - \lambda \cdot u_c)P_c + (\lambda \cdot u_c)S_{\text{text}}^{(c)}$$

Key Contributions¶

Dual-stream MIL architecture that bridges 2D VLMs to 3D volumetric CT without dense annotations, using raw reports as sole supervision
TCA mechanism enabling pathology-specific slice retrieval via cross-modal attention with text queries
UAR inference module that dynamically blends MIL and semantic scores proportional to prediction uncertainty
Asymmetric Loss integration to address severe class imbalance in multi-label intracranial pathology classification
Zero-shot cross-domain transferability demonstrated on external CQ500 dataset without retraining

Results¶

Brain-Adapter on internal clinical dataset (852 NCCT, 21 classes): Micro AUC 0.887, Macro AUC 0.778, Hamming Loss 0.079, Micro SEN 0.865, Micro SPE 0.982
vs. best competing MIL baseline (BiomedCLIP+ABMIL): Micro AUC 0.860 → 0.887, Macro AUC 0.750 → 0.778, Hamming Loss 0.088 → 0.079
vs. 3D volumetric transformers: ViT-B Macro AUC 0.534, Swin-B 0.535—both far below Brain-Adapter's 0.778
ASL vs. BCE: Micro SEN jumps from 0.285 to 0.740, confirming ASL is critical for imbalanced multi-label setting
Zero-shot on CQ500 (491 scans, 10 abnormalities): Brain-Adapter consistently surpasses BiomedCLIP Mean-Pooling baseline across all classes (exact per-class AUCs shown in Fig. 3a, not tabulated in text)
Ablation: each component (TCA $\mathcal{L}_{\text{align}}$, $\mathcal{L}_{\text{cons}}$, UAR) incrementally improves Macro AUC from 0.760 to 0.778

Limitations¶

Internal dataset is small (852 studies, 682 train / 170 test) from a single institution, limiting generalizability assessment
Zero-shot results on CQ500 are presented only as figures without precise numeric comparisons in the text
LoRA adaptation is applied only to vision encoder self-attention; full fine-tuning or adaptation of the text encoder is not explored
UAR hyperparameters ($\alpha$, $\lambda$, $\tau$) require tuning and their sensitivity is not analyzed
Evaluated only on non-contrast CT; contrast-enhanced or multimodal imaging scenarios are not addressed

Relevance to Vision-Language Models¶

Brain-Adapter is a direct application of 2D biomedical VLMs (BiomedCLIP) to the underexplored 3D medical imaging domain, demonstrating that frozen VLM encoders + lightweight LoRA adaptation can outperform purpose-built 3D transformers at a fraction of the annotation cost. The TCA mechanism extends cross-modal attention to act as a semantic retrieval module—query-by-text over spatial instances—which is a broadly applicable pattern for any 3D VLM transfer scenario. The UAR strategy is a practical test-time fusion technique relevant to any VLM deployment where prediction confidence is heterogeneous. This work contributes to the growing evidence that 2D VLM representations, combined with MIL pooling, are a competitive and scalable alternative to native 3D architectures for volumetric understanding.