CUPCase: Clinically Uncommon Patient Cases and Diagnoses Dataset¶

🕒 Published (v1): 2025-03-08 13:21 UTC · Source: Arxiv · Venue: AAAI 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

CUPCase is a benchmark of 3,562 real-world clinical case reports from BMC (2012–2020) designed to evaluate LLMs on complex, atypical diagnoses—rare diseases, uncommon presentations, and unexpected treatment responses. It provides both multiple-choice (with semantically hard distractors) and open-ended free-text diagnosis tasks. Evaluation reveals that general-purpose GPT-4o outperforms all tested clinical LLMs.

Problem¶

Existing medical benchmarks (MedQA, PubMedQA, MedMCQA) derive from medical exam questions or curated literature, testing textbook knowledge rather than real-world diagnostic complexity. They under-represent rare diseases, atypical presentations, and cases deviating from classical patterns—the exact scenarios where clinical decision support is most needed. Prior real-world case report benchmarks were small (≤75 cases) or restricted to a single center or disease category.

Method¶

Case reports were scraped from BMC Journal of Medical Case Reports (2012–2020). GPT-4o-mini was used to (1) extract the "case presentation" section and (2) strip explicit references to the final diagnosis and post-diagnosis treatment via iterative prompt refinement with manual validation on 35-sample subsets. For the multiple-choice variant, the JINA AI embedding model converted each diagnosis into a 768-dimensional vector; cosine similarity against all 94,766 ICD-10-CM long descriptions selected three semantically close distractors (mean BERTScore F1 ~69% between distractor and correct answer, confirming hardness). LLMs were evaluated zero-shot on both tasks: accuracy for multiple-choice (lm-evaluation-harness, token-likelihood selection) and BERTScore F1 for free-text generation. A cumulative token analysis assessed performance with incrementally revealed case text (20%, 40%, … 100%).

Key Contributions¶

3,562 real patient case presentations with final diagnoses in both free-text and ICD-10-CM-grounded multiple-choice format, spanning 28 medical disciplines
Hard distractor generation via ICD-10-CM embedding cosine similarity, validated against semantic overlap
Systematic zero-shot benchmarking of 13 LLMs (5 general-purpose, 8 clinical) on both task formats
Cumulative information analysis simulating progressive information disclosure during diagnosis
Open-source dataset (HuggingFace) and preprocessing code enabling extension to additional journals

Results¶

Multiple-choice accuracy: GPT-4o 87.9% > Llama3-70B Instruct 85.7% > Meditron-70B 85.55% > Llama3.1-70B 84.8%; best small model: Meditron-7B 82.27%; weakest: Meerkat 77.47%
Free-text BERTScore F1: GPT-4o 76.42% > Llama3-70B Instruct 72.67% > Llama3.1-70B 72.31%; best CLLM: MedLM-Large 71.32%; Meditron-70B drops to 59.77% despite leading on MC
General > Clinical: All tested CLLMs underperformed GPT-4o and Llama3-70B on both tasks; Llama3-OpenBioLLM-70B underperformed its base Llama3-70B-Instruct despite medical instruction tuning
Cumulative analysis: GPT-4o retains 87.3% (MC) and 88.6% (free-text) of full-context performance using only the first 20% of case tokens

Limitations¶

Dataset restricted to a single journal (BMC Medical Case Reports); may not represent full clinical diversity
Only ~1% of samples manually validated at each preprocessing step; full dataset not human-reviewed
Zero-shot only; few-shot evaluation omitted due to long context lengths
Image data excluded from evaluation despite being present in case reports
Cannot rule out data contamination: case reports may appear in LLM pretraining corpora

Relevance to Foundation Models in Medicine¶

CUPCase directly stress-tests the clinical generalization of medical foundation models by moving beyond exam-style benchmarks to complex, atypical real-world cases—the regime where domain-specific fine-tuning might be expected to yield the largest gains. The consistent finding that general-purpose LLMs (GPT-4o, Llama3-70B) outperform domain-specialized CLLMs challenges the assumption that medical pre-training or instruction-tuning on clinical text necessarily improves diagnostic reasoning on hard cases. For researchers tracking foundation models in medicine, this benchmark provides a scalable, open evaluation resource for probing genuine clinical reasoning rather than memorized exam answers, and its multimodal structure (text + images) positions it as a future testbed for VLMs. The early-token analysis also surfaces a practical signal for deployment: large general-purpose models provide clinically useful differential diagnoses with very limited initial patient context.