Skip to content

MRI2Rep: Autoregressive Structured Report Generation for 3D Liver MRI

🕒 Published (v1): 2026-06-24 01:31 UTC · Source: Arxiv · Venue: MICCAI 2026 · link

Why this paper was selected

MICCAI; Zongwei Zhou; autoregressive structured MRI reports end-to-end

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MRI2Rep is an autoregressive encoder–decoder framework that generates LI-RADS-structured radiology reports directly from 3D multi-phase liver MRI, without requiring lesion-level spatial annotations. A Report-to-Label Canonicalization (RLC) module converts noisy free-text reports into closed-vocabulary diagnostic sequences, enabling clean supervision from retrospective clinical data. On a 383-case held-out test set, MRI2Rep achieves 76.0% case-level sensitivity and 29.4% lesion-level F1, far outperforming adapted 3D vision–language baselines (≤8.3% F1).

Problem

End-to-end structured report generation for 3D liver MRI is underexplored. Two compounding challenges block progress: (1) free-text report supervision is noisy due to heterogeneous phrasing, negation, and uncertainty—especially dangerous for small 3D cohorts where overfitting is a major risk; (2) 3D volumetric inputs impose large computational and representational burdens, requiring models to capture subtle cross-sequence cues (e.g., washout dynamics across arterial and portal-venous phases) and long-range 3D spatial relationships. No prior system enforces LI-RADS decision rules or produces closed-vocabulary structured output without lesion-level bounding-box or segmentation annotations.

Method

MRI2Rep operates in three sequential stages:

Stage 1 — Report-to-Label Canonicalization (RLC): An LLM (Claude-3.5 / GPT-5 / DeepSeek-R1) receives LI-RADS major-feature decision logic with chain-of-thought prompting and converts each free-text report \(R\) into a structured sequence \(y = (y_1, \ldots, y_T)\) of lesion triplets \(\{(y_k^{\text{type}}, y_k^{\text{pos}}, y_k^{\text{qty}})\}\) plus a liver-background token, over a shared vocabulary of \(|V|=24\) tokens. Each triplet is evidence-grounded by a verbatim sentence \(e_k \subset R\); abstention when evidence is absent makes labels reproducible and auditable.

Stage 2 — Autoregressive Encoder–Decoder: The visual encoder \(f_\theta\) is a 3D CNN (four Conv3d-IN-GELU blocks, stride-2 downsampling to 256-d visual tokens) followed by a visual Transformer for global 3D self-attention, producing memory tokens \(E_\text{vis} \in \mathbb{R}^{S_\text{vis} \times d}\). The input volume \(V \in \mathbb{R}^{3 \times H \times W \times D}\) stacks arterial-phase (ART), portal-venous-phase (PV), and a binary liver mask. An autoregressive Transformer decoder cross-attends to \(E_\text{vis}\) and predicts tokens sequentially. An auxiliary binary lesion-presence head (global average pooling → linear classifier) provides an extra discriminative signal. The joint loss is: $\(\mathcal{L} = \mathcal{L}_\text{CE}(y, \hat{y}) + \lambda \mathcal{L}_\text{BCE}(z, \hat{z})\)$ with class-reweighted cross-entropy and label smoothing. The CNN backbone is pre-trained to reconstruct PV from ART (MSE + SSIM) before fine-tuning. Word dropout randomly replaces decoder input tokens with PAD to reduce text-prefix reliance.

Stage 3 — Template Rendering: Predicted token sequences are deterministically mapped to natural-language sentences via hand-crafted templates with paraphrase pools generated by Claude-3.5, preventing lexical artifacts that could unblind radiologists.

Key Contributions

  • A dataset of 3,929 real-world liver MRI–report pairs (3,830 with complete multi-phase sequences) from a 10-year single-institution cohort, with canonicalized RLC labels.
  • RLC: a LI-RADS-guided LLM prompting protocol converting free-text reports to auditable, closed-vocabulary structured sequences with evidence grounding; raises radiologist label-retention from ~77–84% to ≥92% across all categories and three LLM backends.
  • First end-to-end LI-RADS-structured reporting system for 3D liver MRI operating without lesion-level spatial annotations.
  • LLM-Eval validated as a conservative automated judge: Claude-3.5 accepts 61.8% of AI reports vs. 75%/70% for human radiologists, with ≥98% label-extraction accuracy confirming reliability as a high-throughput proxy.

Results

  • Case-level sensitivity: 76.0% (vs. best baseline RadFM 21.6%, Merlin 25.1%)
  • Lesion-level F1: 29.4% (vs. best baseline RadFM 8.3%—a >21-point margin)
  • Triplet-F1 (type+pos+qty): 28.1%
  • Liver-Acc: 82.4% (vs. Multi-label cls. 79.6%)
  • LLM-Eval: 61.8% (vs. Multi-label cls. 33.4%)
  • Clinical reader study (n=100): 75%/70% AI-report acceptance by junior/senior radiologists vs. 95%/100% for originals; Cohen's Îş = 0.97
  • Ablation findings:
  • Removing PV phase: Lesion-F1 drops from 29.4 → 13.5 (largest single ablation)
  • Removing ART-to-PV pre-training: Lesion-F1 drops to 22.1
  • Removing visual self-attention: Lesion-F1 drops to 10.4
  • Multi-label classification baseline: Lesion-F1 14.8% (confirms autoregressive decoding captures inter-lesion dependencies)
  • Per-class F1 range: 51.7% (cysts) to 11.3% (targetoid lesions); multi-lesion cases drop from 38.4% to 21.7% F1

Limitations

  • Report-only supervision without spatial annotations limits lesion-level F1, particularly for multi-lesion sequences and post-treatment cases (34.1% of false positives arise in prior-ablation cases).
  • Closed RLC vocabulary (24 tokens) bounds expressible semantics; findings outside LI-RADS categories or requiring nuanced description cannot be captured.
  • Single-centre data (one academic hospital, Siemens scanners, extracellular gadolinium) leaves cross-centre and cross-scanner generalisation unvalidated.
  • LI-RADS categories formally apply only to at-risk HCC patients; non-HCC tokens extend coverage heuristically.
  • k-fold cross-validation is impractical for this 3D autoregressive pipeline; the stratified set-cover split mitigates but does not eliminate split variance.

Relevance to Vision-Language Models

MRI2Rep extends the 3D medical VLM literature—which has largely targeted representation learning, classification, or VQA—to the harder task of structured, guideline-conformant text generation from volumetric inputs, demonstrating that domain-constrained closed vocabularies (LI-RADS, 24 tokens) with auditable LLM-based label canonicalization can dramatically reduce the noisy-supervision problem that plagues small 3D datasets. The RLC design pattern—using a strong LLM to canonicalize free-text into structured supervision rather than training end-to-end on raw text—is broadly transferable to other medical imaging domains where annotation is scarce and terminology is standardized. The validated LLM-as-judge methodology (Claude-3.5 as conservative automated evaluator) is directly applicable to other radiology VLM evaluation pipelines. The ≥21-point Lesion-F1 gap over adapted 3D VLM baselines (M3D, CT2Rep, Merlin, RadFM) quantifies the cost of applying generic 3D vision–language models to domain-specific structured-reporting tasks.