Skip to content

Multi-LLM Collaborative MRI Report Generation for Visual Instruction Tuning in Brain Oncology

🕒 Published (v1): 2026-07-16 05:21 UTC · Source: Arxiv · link

Why this paper was selected

Multi-LLM VLM pipeline for 3D brain MRI reports; novel visual instruction tuning for oncology

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Existing medical VLMs are bottlenecked by the absence of 3D image-text paired datasets, constraining them to 2D modalities like chest X-rays. This paper constructs a 3D brain oncology MRI-text dataset via a multi-LLM collaborative pipeline (GPT-4o-mini generation + three-LLM majority-vote refinement) and trains a VLM using VQ-GAN tokenization and LoRA fine-tuning on Vicuna-7B. The resulting model substantially outperforms both 2D and existing 3D baselines on report generation and VQA tasks.

Problem

No large-scale paired 3D image-text datasets exist for training medical VLMs, forcing the field to remain anchored to 2D imaging. Expert annotation is expensive, slow, and subjective. Prior 3D VLMs (e.g., M3D) were designed for CT, not MRI, and fail to generalize. Brain oncology specifically requires full volumetric context that a single axial slice cannot provide.

Method

Step 1 — Instruction-driven data generation. For each subject, tumor location is mapped to one of seven anatomical atlas regions; tumor shape is characterized by radiomics features (volume, sphericity). This structured metadata plus modality/disease label is fed with a criteria-enforcing instruction prompt to GPT-4o-mini, which generates an initial free-text radiology report and a closed-ended VQA set.

Step 2 — Multi-LLM collaborative refinement. Three independent LLM reviewers (Claude 3.5 Sonnet, Gemini 1.5 Flash-8B, DeepSeek-chat) each evaluate the initial report on two criteria: accuracy of information integration and consistency with clinical knowledge. Majority-reject (≤1 acceptance) triggers regeneration by GPT-4o-mini incorporating the reviewers' feedback; majority-accept adds the report to the refined dataset.

Step 3 — Visual instruction tuning. 3D MRI volumes are encoded into latent representations by a frozen pretrained VQ-GAN encoder: $\(Z_i = E_{VQ}(X_i), \quad T_i = \text{Flatten}(\text{Conv3D}(Z_i))\)$ A trainable 3D Perceiver projects these into image tokens. Training is two-stage on Vicuna-7B-1.5: (1) 5 epochs with only the 3D Perceiver trained, LLM frozen; (2) 5 epochs jointly fine-tuning the Perceiver and the LLM with LoRA (\(r=32\), \(\alpha=32\), lr \(= 10^{-6}\), AdamW, single H100).

Key Contributions

  • Novel 3D brain oncology image-text dataset built from BraTS2021-GLI (1251 glioma subjects) and BraTS2023-MEN (1000 meningioma subjects), totaling 9004 3D volumes, with no manual text annotation.
  • Multi-LLM majority-vote refinement pipeline that replaces costly expert validation and measurably increases inter-cluster separation of tumor-type and modality embeddings in the refined vs. initial reports (t-SNE centroid distance: 39.81 → 53.13).
  • End-to-end 3D VLM (VQ-GAN + 3D Perceiver + LoRA-tuned Vicuna-7B) for simultaneous free-text report generation and closed-ended VQA on brain MRI.

Results

Report generation (higher is better; bold = proposed method):

  • BraTS2021-GLI: BLEU 11.72, ROUGE 43.57, METEOR 33.59, BERT-F1 80.61 vs. best 2D baseline LLaVA (ROUGE 18.97, BERT-F1 69.57) and best 3D baseline M3D-LLaMA2 (ROUGE 9.65, BERT-F1 66.17).
  • BraTS2023-MEN: BLEU 9.76, ROUGE 42.56, METEOR 33.18, BERT-F1 79.29 vs. LLaVA (ROUGE 18.38, BERT-F1 69.44) and M3D-LLaMA2 (ROUGE 9.58, BERT-F1 66.30).

VQA (tumor type / modality / location, multiple-choice):

  • BraTS2021-GLI: Accuracy 91.69%, F1 79.39% vs. best 2D ChatGPT (53.22%, 52.24%) and best 3D M3D-LLaMA2 (50.83%, 35.47%).
  • BraTS2023-MEN: Accuracy 88.95%, F1 84.03% vs. ChatGPT (39.02%, 49.40%) and M3D-LLaMA2 (29.51%, 23.82%).

Limitations

  • No clinical validation by expert radiologists; NLP metrics (BLEU, ROUGE, BERT-F1) cannot confirm diagnostic correctness.
  • Self-consistency bias: both training and evaluation ground truth are LLM-generated, so performance gains may partly reflect synthetic-data alignment rather than genuine diagnostic capability.
  • Evaluation is confined to curated BraTS datasets with clean labels and well-defined segmentation masks; generalization to noisy real-world clinical acquisitions is unverified.
  • No external test set independent of the LLM-generated data pipeline; model performance against real radiologist reports is unknown.

Relevance to Vision-Language Models

This paper directly addresses the 2D-to-3D scalability bottleneck in medical VLMs by proposing a fully automated, annotation-free pipeline for constructing 3D image-text training data — a significant infrastructure contribution for the field. The multi-LLM majority-vote refinement strategy offers a generalizable quality-control mechanism that could substitute for human annotation in other data-scarce medical domains. The VQ-GAN + 3D Perceiver + LoRA architecture demonstrates a concrete design pattern for lifting existing 2D LLaVA-style frameworks to volumetric input without full retraining. Researchers tracking VLM grounding, medical multimodal models, or synthetic data pipelines for instruction tuning will find both the dataset construction methodology and the architecture adaptation directly relevant.