Skip to content

Are Video Models Emerging as Zero-Shot Learners and Reasoners in Medical Imaging?

🕒 Published (v1): 2025-10-11 15:19 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper investigates whether a general-purpose autoregressive Large Vision Model (LVM), trained exclusively on natural image/video data, can perform medical imaging tasks zero-shot. Across organ segmentation, denoising, super-resolution, and 4D CT motion prediction, LVM achieves competitive performance—and surpasses all specialized baselines on motion prediction—without any medical-domain fine-tuning.

Problem

Medical imaging relies on fragmented, task-specific pipelines (separate models for segmentation, registration, motion prediction), and no unified framework generalizes across these tasks. Existing medical foundation models handle static images but lack explicit temporal modeling for dynamic data (4D CT, cine MRI, ultrasound video). The question is whether large-scale video pretraining implicitly acquires spatiotemporal priors transferable to medical domains without retraining.

Method

The authors adopt LVM (Bai et al., CVPR 2024), a 120M-parameter decoder-only LLaMA-style Transformer pretrained on natural images/videos via autoregressive next-token prediction. Medical CT volumes are tokenized using a VQGAN (f=16 downsampling, 8192-codebook), compressing each 256Ă—256 CT slice into 256 discrete tokens arranged in a 16Ă—16 grid. For 4D CT motion prediction, temporal sequences of CT phases are concatenated as visual "sentences" and the model autoregressively predicts future phase tokens conditioned on all prior observed (and previously predicted) phases. For evaluation, nnUNet (TotalSegmentator weights) extracts organ masks from both predicted and ground-truth CT volumes to compute spatial overlap metrics. An ablation tests three input modalities: CT only, segmentation mask only, and CT+Mask (default). The same model is also evaluated zero-shot on segmentation (text-prompted), denoising (synthetic noise), and super-resolution tasks with no task-specific adaptation.

Key Contributions

  • First systematic evaluation of a zero-shot general-purpose video model across four medical imaging task categories (segmentation, denoising, super-resolution, motion prediction) without any medical training data.
  • Demonstration that autoregressive sequence modeling of 4D CT phases enables state-of-the-art organ motion prediction, outperforming DVF-supervised and generative baselines across lung, heart, and liver.
  • Evidence of emergent temporal reasoning: the model maintains DSC >85% across five predicted phases with less error accumulation than all baselines, attributed to full-history autoregressive conditioning.
  • Ablation showing CT+Mask input is complementary: texture provides deformation cues; masks constrain learning to anatomically relevant regions.

Results

  • Motion prediction (public dataset, lung): LVM IoU 90.75%, DSC 95.15% vs. best baseline RMSim IoU 84.05%, DSC 89.16% (+6.7% IoU, +6.0% DSC).
  • Motion prediction (private dataset, liver): LVM IoU 91.99%, DSC 95.83% vs. RMSim IoU 83.05%, DSC 88.08% (+8.7% IoU, +7.0% DSC).
  • Motion prediction (private dataset, heart): LVM IoU 87.83%, DSC 93.24%, SD 3.34 px, HD95 5.38 px, all best in class.
  • Multi-phase prediction: LVM maintains DSC >85% at phase #5; all baselines show steeper decline.
  • Zero-shot segmentation: DSC 91.52%, IoU 87.12%.
  • Zero-shot denoising: PSNR 34.44 dB, SSIM 0.8538.
  • Zero-shot super-resolution: PSNR 29.56 dB, SSIM 0.7835.
  • Evaluated on 122 patients, 1,820+ 3D CT volumes across three datasets (public NSCLC, private lung/heart, private liver HCC).

Limitations

  • Performance on segmentation, denoising, and super-resolution does not match fully supervised or fine-tuned domain-specific models.
  • Only intra-fraction (single-session) motion modeled; inter-fraction changes (tumor shrinkage, weight loss over treatment weeks) are not addressed.
  • Detection task is mentioned but no quantitative results are reported in the paper.
  • VQGAN was fine-tuned on 4D CT data, so the pipeline is not entirely training-free for the tokenizer component.
  • Evaluation restricted to CT modality; generalization to MRI, ultrasound, or other dynamic modalities is asserted but not demonstrated.
  • Clinical validation absent; no assessment of downstream impact on radiotherapy planning margins.

Relevance to Foundation Models in Medicine

This work is a direct probe of the hypothesis that video foundation model pretraining—rather than medical-specific pretraining—may be sufficient to bootstrap general-purpose medical visual reasoning, which has major implications for the field's pretraining strategy. It complements efforts like MedGemma and Med-Flamingo by showing that modality-agnostic temporal pretraining can transfer to dynamic medical tasks (4D CT) where existing medical VLMs lack temporal modeling. The LVM's outperformance of DVF-supervised specialist models on motion prediction without any medical data is a striking data point for scaling-law advocates arguing for unified, domain-agnostic foundation models. Practically, it motivates building next-generation medical foundation models on video architectures and raises the question of how much medical-domain pretraining is actually needed versus what emerges from large-scale natural video pretraining alone.