Skip to content

OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysis

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; unified slice-volume LVLM covering full CT image analysis pipeline

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OmniCT is a unified large vision–language model (LVLM) for CT imaging that jointly handles 2D slices and 3D volumes within a single framework. It introduces spatial consistency and organ-level semantic enhancements to overcome the longstanding dichotomy between slice-driven and volume-driven medical LVLMs. It also releases MedEval-CT, the largest CT VQA dataset (1.7M samples) and benchmark to date.

Problem

Existing medical LVLMs are siloed: slice-driven models lack cross-slice spatial consistency, while volume-driven models are coarse-grained and incompatible with slice inputs. No unified modeling paradigm exists that simultaneously satisfies fine-grained 2D local sensitivity (sub-centimeter nodules, lesion boundaries) and macro-level 3D volumetric reasoning (tumor infiltration, inter-organ topology). Standard medical benchmarks also lack CT-specific task alignment.

Method

OmniCT adds two modules on top of a standard vision-encoder + LLM backbone:

Spatial Consistency Enhancement (SCE): - Volumetric Slice Composition (VSC): Adjacent slices along \(z\) are concatenated channel-wise into units \(\hat{s}_i = \text{Concat}(V_{3i-2}, V_{3i-1}, V_{3i}) \in \mathbb{R}^{3 \times H \times W}\), unifying 2D slices (replicated) and 3D volumes into the same \(3 \times H \times W\) format. - Tri-Axial Positional Embedding (TPE): Sinusoidal encodings along depth \(N_s\), height \(H'\), and width \(W'\) are concatenated to patch tokens: \(Z = F \oplus P^{N_s} \oplus P^{H'} \oplus P^{W'}\). - MoE Hybrid Projection (MHP): A token-level unshuffle compresses \(m \times m\) spatial neighbors; a Mixture-of-Experts projector with slice-specific \(W_s\), volume-specific \(W_v\), and shared \(W_\text{share}\) weights maps features to LLM space: \(\hat{F} = W_\text{share}\,\sigma(W_s \hat{Z} \cdot \mathbf{1}_\text{slice} + W_v \hat{Z} \cdot \mathbf{1}_\text{volume})\).

Organ-level Semantic Enhancement (OSE): - TotalSegmentor generates 117-class organ masks \(M_o\) which index organ-specific token subsets \(\hat{F}_o \in \mathbb{R}^{L_o \times d_h}\). - An adaptive aggregation function compresses \(\hat{F}_o\) to fixed length \(L_c\), applying a magnification effect for small organs and a compression effect for large ones. - Global tokens \(\hat{F}\) and organ tokens \(\hat{f}_o\) are concatenated: \(\hat{F}_\text{OSE} = [\hat{F};\, \hat{f}_o]\).

Training uses two stages: pretraining (MHP only) and instruction tuning (MHP + LLM), optimizing autoregressive cross-entropy loss.

Key Contributions

  • Unified slice–volume paradigm: single LVLM handles both 2D CT slices and 3D CT volumes without architecture switching.
  • SCE module: VSC + TPE + MHP inject volumetric spatial priors while remaining compatible with slice inputs.
  • OSE module: organ-mask-guided token selection with adaptive aggregation embeds clinically relevant anatomical semantics.
  • MedEval-CT: 1.7M VQA samples from 170,280 3D volumes and 327,063 2D slices across 7 task types, 4 clinical difficulty levels, and 13 organ classes; accompanied by a standardized benchmark (MedEval-CT-Bench) and evaluation toolkit (MedEval-CT-Factory).

Results

  • Slice benchmarks (Table 2): OmniCT-7B achieves average score 81.45, exceeding the second-best model (Lingshu-7B) by +11.01 points; OmniCT-3B reaches 77.71.
  • Volume benchmarks (Table 3): OmniCT-3B reaches 87.38 on CT-RATE multi-choice (average 63.48); OmniCT-7B reaches 85.69 on the 3D-RAD LTD task (average 66.15), outperforming all compared models including CT-CHAT, M3D-LaMed, GPT-5.
  • Ablation (Table 1): Adding SCE raises the MedEval-CT average from 77.62 → 78.06; adding OSE alone → 78.62; both together → 79.62; gains are consistent across 2D, 3D, organ, and task sub-axes.
  • CT-CHAT achieves 86.46 on CT-RATE multi-choice but overall average below 36, demonstrating instability without the unified paradigm.

Limitations

  • OSE depends on TotalSegmentor segmentation masks at inference time, introducing a preprocessing dependency and potential failure modes when segmentation quality is poor.
  • Results on CT report generation for volume tasks are comparable to (not clearly better than) models specifically designed for CT volume report generation.
  • Evaluation is restricted to CT; generalizability to other volumetric modalities (MRI, PET) is not demonstrated.
  • The 3-slice grouping in VSC is a fixed heuristic; performance on very thin-slice or anisotropic acquisitions is not analyzed.

Relevance to Vision-Language Models

OmniCT directly addresses one of the core architectural challenges in medical VLMs: the modality gap between 2D and 3D visual inputs that forces practitioners to maintain separate model families. The MoE-based projection and tri-axial positional encoding offer a practical blueprint for extending general VLM architectures (InternVL, Qwen-VL) to volumetric inputs without abandoning 2D pretraining. MedEval-CT establishes a standardized, large-scale CT benchmark that fills a gap analogous to what VQAv2 or MMMU fill for general VLMs, enabling fairer cross-model comparisons. The organ-mask grounding strategy (OSE) also illustrates a principled approach to spatially grounded VLMs in domains where precise region attribution is clinically required.