Skip to content

TMF-RSE: Tri-Modal Fusion with Regional Semantics and Evidential Uncertainty for Lung Severity Scoring

🕒 Published (v1): 2026-07-07 14:54 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TMF-RSE is a tri-modal deep learning framework for continuous lung severity scoring that fuses appearance features (DINOv3-ViT), structural features (SAM-derived segmentation masks), and clinical semantic features (frozen LLaVA-Med VLM) via semantic gating, structural prior modulation, and hierarchical fusion. An evidential regression head outputs Normal-Inverse-Gamma parameters to yield both severity predictions and calibrated aleatoric/epistemic uncertainty estimates. The model achieves state-of-the-art MAE and Pearson correlation on the Per-COVID-19 CT and RALO chest radiograph benchmarks.

Problem

Existing transformer-based lung severity regressors (ViTReg-IP, MViTReg-IP, QCross-Att-PVT, PViTGAtt-IP) exploit appearance cues only, without incorporating explicit structural priors from lung segmentation or grounded clinical semantics from medical VLMs. Additionally, few severity systems combine evidential uncertainty with tri-modal fusion and representation-level structural confidence, leaving calibrated uncertainty estimation largely unaddressed.

Method

Three frozen/lightweight encoders extract complementary representations: 1. Image encoder: DINOv3-ViT-S/16 (self-supervised ViT) produces 384-dim features projected to 512-dim. 2. Mask encoder: Lightweight CNN processes SAM3-generated binary lung masks, capturing anatomical boundaries; output projected to 256-dim. 3. VLM encoder: Frozen LLaVA-Med v1.5-Mistral-7B processes upper, middle, and lower lung zone crops independently with region-specific clinical prompts, producing a \(3 \times 768\) semantic tensor.

These are integrated by three complementary fusion mechanisms: - Semantic gating: VLM features generate sigmoid gates \(g = \sigma(\text{MLP}(f_\text{vlm}))\) applied element-wise to image features: \(f_\text{img}^\text{gated} = f_\text{img} \odot g\). - Structural prior modulation: Mask-derived structural confidence \(c_\text{struct} = \sigma(\text{MLP}(f_\text{mask}))\) blends attention weights toward uniform when boundaries are ill-defined: \(a_\text{mod} = c_\text{struct} \cdot a + (1 - c_\text{struct}) \cdot a_\text{uniform}\). - Hierarchical fusion: Three-stage progressive integration — (1) image+mask → structural-appearance, (2) + VLM → semantically-aware structural, (3) synergy gate over all three — supplemented by pairwise cross-attention (8 heads).

An evidential regression head outputs NIG parameters \((\gamma, \nu, \alpha, \beta)\); severity is predicted as \(\gamma\) and uncertainties as \(\text{Aleatoric} = \frac{\beta}{\alpha-1}\), \(\text{Epistemic} = \frac{\beta}{\nu(\alpha-1)}\). Loss = NLL + \(\lambda \cdot \text{Reg}(\nu, \alpha)\) with \(\lambda = 0.001\).

Key Contributions

  • Tri-modal architecture unifying appearance (DINOv3), structural (SAM masks), and semantic (LLaVA-Med) features under semantic gating, structural prior modulation, and hierarchical fusion.
  • Region-specific VLM prompting over three lung zones (upper/middle/lower) as a frozen semantic prior, enabling spatially-grounded clinical semantics without fine-tuning.
  • Dual-level uncertainty modeling: structural prior modulation at representation level + evidential NIG regression at output level.
  • Modality ablation across image-only, image+mask, image+VLM, and full tri-modal configurations on two benchmarks.

Results

RALO (geographic extent): - TMF-RSE: MAE 0.339, PC 0.973 vs. best prior PViTGAtt-IP: MAE 0.351, PC 0.963

RALO (lung opacity): - TMF-RSE: MAE 0.310, PC 0.960 vs. best prior PViTGAtt-IP: MAE 0.322, PC 0.941

Per-COVID-19 CT validation phase: - TMF-RSE: MAE 4.02, PC 0.9629, RMSE 7.07 vs. best prior PViTGAtt-IP: MAE 4.09, PC 0.9439; best challenge entry (TAC): MAE 4.48, PC 0.9460

Per-COVID-19 CT test phase: - TMF-RSE: MAE 3.69, PC 0.9122, RMSE 6.15 vs. best prior PViTGAtt-IP: MAE 4.45, PC 0.8094; best challenge entry (Taiyuan/TAC): MAE 3.55

Modality ablations (Per-COVID-19 CT val): - Image only: MAE 4.82, PC 0.9436 - Image+mask: MAE 4.67, PC 0.9498 - Image+VLM: MAE 4.53, PC 0.9517 - Full TMF-RSE: MAE 4.02, PC 0.9629

Uncertainty sparsification: progressively removing highest-uncertainty samples monotonically reduces MAE on the retained subset, validating evidential scores as difficulty indicators.

Limitations

  • Evaluated only on Per-COVID-19 CT and RALO; no external validation on held-out hospitals, scanner types, or non-COVID etiologies.
  • Frozen LLaVA-Med forces batch size 1 (NVIDIA L4 GPU), with training times of 8–12 hours per variant; computationally expensive.
  • Zone division into three equal vertical bands is coarse and may not align with anatomical landmarks in severe disease.
  • Benchmark label definitions (radiologist-estimated vs. segmentation-derived on Per-COVID-19; continuous GE/LO on RALO) may not capture all clinical nuances.
  • No clinical validation confirming whether uncertainty estimates correlate with patient outcomes or improve clinical decision-making.
  • Single-region vs. multi-region VLM prompt comparison is absent from the main tables.

Relevance to Vision-Language Models

This paper demonstrates a concrete pattern for deploying a frozen medical VLM (LLaVA-Med) as a semantic prior inside a discriminative regression pipeline rather than as a generative end-to-end model, sidestepping fine-tuning costs while injecting grounded clinical language into image-only regressors. The region-specific prompting strategy — decomposing the image spatially and querying the VLM per zone — is a transferable technique for any VLM-augmented medical imaging task where lesion distribution is spatially heterogeneous. For VLM researchers, the semantic gating mechanism provides an interpretable interface between VLM embeddings and visual feature selection, and the results quantify (via ablation) the incremental gain of VLM semantics over structure-only fusion. This complements the broader line of work on BiomedCLIP, MedCLIP, and MedGemma by showing how frozen VLM encoders can improve continuous regression rather than classification or report generation.