Skip to content

Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoning

🕒 Published (v1): 2025-09-23 14:42 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Citrus-V is a unified multimodal medical foundation model that extends the authors' prior text-only Citrus system to handle pixel-level visual grounding by integrating object detection, image segmentation, and multimodal chain-of-thought (CoT) reasoning within a single framework. It is accompanied by a curated open-source data suite spanning reasoning, detection, segmentation, and document understanding. Evaluations show Citrus-V surpasses existing open-source medical models and specialist imaging systems across multiple benchmarks.

Problem

Existing medical AI systems are narrow expert models requiring separate networks per organ, modality, or task (detection, segmentation, classification, report generation), and cannot generalize across clinical scenarios. Multimodal large models lack pixel-level visual grounding and integrated chain-of-thought reasoning, limiting clinical utility for structured diagnosis, lesion quantification, and second opinions.

Method

Citrus-V is built on the prior Citrus text-only medical LLM and adds vision capabilities via full-parameter supervised fine-tuning (not LoRA, to avoid limiting multi-task synergy). A novel multimodal curriculum training paradigm is introduced to accelerate convergence across heterogeneous tasks.

Four data streams are synthesized and curated: 1. Medical Visual Reasoning: A dual-model pipeline uses a generation LLM to produce VQA pairs from complex, long-answer-requiring queries, then an answering LLM to annotate step-by-step reasoning paths; only pairs with consistent reasoning→answer chains are retained. 2. Medical Document Understanding: ~2M real patient-uploaded document images (Chinese online consultations) are processed through geometry normalization (rotation/dewarping), dual-VLM table parsing, and LLM agreement checking to produce QA pairs across Lab Test Reports (LTR) and General Medical Documents (GMD). Hard cases (requiring normalization to reach consensus) are prioritized in post-training. 3. Detection and Segmentation Annotation: Two automatic pipelines — (i) off-the-shelf nnU-Net/SegFormer3D with tiered expert review; (ii) VoxelMorph registration-based label transfer from template to target CT, extending to rare anatomy — slash manual annotation by ~10×. 4. Multimodal CoT for Chest Radiology: CT volumes are DRR-projected to synthesized X-rays, paired with CT reports converted to X-ray-compatible reports by LLM. A YOLO-based bounding box detector trained on projected boxes then enables CoT report generation in <think>...<bbox>...</bbox>...</think><answer> format, where each reasoning step is grounded to a spatial location.

Key Contributions

  • Unified single-model pipeline integrating detection, segmentation, and multimodal CoT reasoning for pixel-level lesion localization, structured report generation, and physician-like diagnostic inference.
  • Open-source data suite: multimodal CoT reasoning dataset for radiology, a refined detection/segmentation benchmark with corrected labels, and MedDocBench (500-item graded-difficulty medical document understanding benchmark).
  • Novel multimodal curriculum training paradigm with full-parameter fine-tuning for deeper vision-language integration.
  • Scalable annotation pipeline combining public model auto-segmentation with registration-based label transfer, reducing manual annotation from months to days.

Results

The paper text provided is truncated before the experimental results section. The abstract claims Citrus-V "outperforms existing open-source medical foundation models and expert-level imaging systems across multiple benchmarks," but no specific numbers, dataset names, or baseline comparisons are present in the provided text.

Limitations

  • Results section is absent from the provided text; specific performance numbers cannot be verified.
  • Medical document understanding data is drawn from Chinese online consultations, raising questions about cross-lingual/cross-institution generalizability.
  • Multimodal CoT synthesis pipeline targets chest radiography (CT→X-ray projection); coverage of other modalities and anatomies is not detailed.
  • The model remains 2D-centric; 3D imaging challenges (quadratic sequence complexity, scarce paired report data) are acknowledged but not resolved.
  • Registration-based annotation relies on deformable registration quality, which degrades for large morphological differences from template.
  • Hard-example mining for document understanding depends on dual-VLM agreement as a proxy for label quality, which may still systematically fail on certain document types.

Relevance to Foundation Models in Medicine

Citrus-V directly targets the central open problem in medical foundation models: bridging pixel-level visual understanding with structured clinical reasoning in a single generalist system, rather than chaining specialist models. Its full-parameter fine-tuning strategy and multimodal CoT grounding (bounding box–anchored reasoning steps) represent a concrete architectural direction for interpretable radiology AI. The accompanying open-source data suite — particularly the CoT reasoning dataset and MedDocBench — fills gaps that have historically forced practitioners to rely on proprietary resources. For researchers tracking this space, the registration-based annotation pipeline and dual-VLM consensus filtering are transferable techniques for scalable dataset creation across other modalities.