Safety and accuracy follow different scaling laws in clinical large language models

🕒 Published (v1): 2026-05-05 17:57 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Scaling clinical LLMs — by model size, RAG, or inference-time compute — does not reliably improve safety. The authors introduce SaFE-Scale, a framework with the RadSaFE-200 radiology benchmark, and show that curated clinician-written evidence is the only intervention that simultaneously improves accuracy and all four safety metrics. Safety and accuracy obey distinct scaling laws.

Problem

Existing clinical LLM evaluation treats all wrong answers equivalently and reports mean accuracy as the primary endpoint. This obscures clinically structured failure modes — high-risk errors, evidence-contradicting answers, and dangerous overconfidence — whose sparse occurrence and asymmetric clinical consequences are invisible to standard benchmarks. The implicit assumption that higher accuracy implies safer behavior is untested and, as shown here, incorrect.

Method

The authors construct RadSaFE-200, a 200-question multiple-choice radiology benchmark with clinician-written clean evidence, conflict evidence, and option-level labels for four safety axes: high-risk error, unsafe answer, evidence contradiction, and dangerous overconfidence (≥0.80 entropy-normalized confidence on a high-risk wrong answer). They evaluate 34 open-weight LLMs (0.5B–685B parameters; Qwen, Llama, Gemma, MedGemma, DeepSeek, Mistral, OpenAI-OSS) under six deployment conditions: closed-book, clean evidence, conflict evidence, standard RAG (Radiopaedia corpus), agentic RAG (the RaR multi-step retrieval-and-reasoning framework), and max-context prompting. Secondary experiments probe 8-sample self-consistency and fixed three-model majority-vote ensembles. A two-way variance decomposition across the 34×6 grid partitions variance in each metric into condition, family, interaction, and residual components.

Key Contributions

• SaFE-Scale framework: systematic protocol for measuring clinical LLM safety across model scale, evidence quality, retrieval strategy, context exposure, and inference-time compute.
• RadSaFE-200 benchmark: 200 radiology MCQs with per-option clinician-defined safety labels, clean evidence, and conflict evidence.
• Empirical decoupling finding: accuracy and safety metrics respond differently to the same deployment changes; they do not follow a single trade-off curve.
• Variance decomposition: deployment condition explains 38–45% of metric variance; model family explains only 8–17%.
• Confidence as a non-signal: confidence on high-risk wrong answers is statistically indistinguishable from confidence on correct answers (gap median well below 30 pp in all conditions).
• Ensemble synchronized failure characterization: majority-vote ensembles often fail unanimously on the same questions; they do not exceed the safest individual member.

Results

• Clean evidence raised mean accuracy from 73.5% (closed-book) to 94.1% (+20.6 pp) while reducing high-risk error 12.0→2.6%, contradiction 12.7→2.3%, dangerous overconfidence 8.0→1.6%. All 34/34 models improved on both axes.
• Standard RAG: accuracy 76.0%, high-risk error 9.6%, contradiction 11.7% — minimal improvement over closed-book.
• Agentic RAG: accuracy 78.1% (+2.1 pp over standard RAG), but dangerous overconfidence remained at 8.0% (same as closed-book) and high-risk error rose to 10.3%; 18/34 models worsened on dangerous overconfidence vs. closed-book.
• Max-context prompting: accuracy 74.0% (near closed-book), high-risk error 10.6%, latency increased from 18.0 s to 27.0 s.
• Conflict evidence: accuracy 92.5% (−1.6 pp vs. clean), high-risk error 3.5%, dangerous overconfidence 2.3% — graded degradation from a single distracting statement.
• Self-consistency (8 samples): accuracy gain +0.4–0.6 pp, high-risk error change −0.4 to +0.1 pp; no meaningful safety improvement across 24 (model, condition) pairs.
• Ensembles: improved panel-average accuracy (closed-book: 73.5→83.0, high-risk error 12.0→7.4) but did not consistently beat the strongest individual member; mean change vs. best member was −0.88 pp accuracy, −0.79 pp high-risk safety.
• Variance decomposition: condition explains 43/45/38% of variance in accuracy/high-risk/dangerous overconfidence; model family explains 9/8/17%.
• Within-family high-risk s.d.: collapses from 3.8 pp (closed-book) to 1.0 pp (clean evidence), independent of model scale.

Limitations

• RadSaFE-200 is small (200 questions) and restricted to radiology; generalization to other medical specialties is unverified.
• Only open-weight locally deployed models are tested; proprietary API-only models (e.g., GPT-4o, Gemini) are absent.
• Clean evidence is clinician-written and curated — operationalizing this in real deployment pipelines is non-trivial and the authors do not specify a practical pathway.
• Self-consistency experiments use only 8 samples on a subset of 8 models; larger sample counts or broader model coverage could show different behavior.
• Dangerous overconfidence metric requires repeated sampling and is undefined for single greedy inference, creating an asymmetry in comparisons.
• Worst-case and synchronized ensemble failure analysis is descriptive; no causal mechanism for question-level concentration of errors is proposed.
• Latency measurements reflect specific local hardware configurations and are not generalizable.

Relevance to Foundation Models in Medicine

This paper directly challenges the dominant paradigm in clinical FM deployment — that scaling model size, context, or retrieval automatically improves safety — and provides the first systematic quantification of how much each scaling axis contributes to safety vs. accuracy independently. For researchers tracking medical foundation models, the core finding (deployment condition explains ~4–5× more variance than model family) reframes evaluation priorities: benchmark accuracy on closed-book MCQs is an unreliable safety proxy. The RadSaFE-200 benchmark and SaFE-Scale framework provide reusable infrastructure for auditing FM safety across deployment conditions, which is directly relevant to regulatory and clinical translation questions. The evidence-quality finding also has implications for RAG-based medical FM systems, since standard retrieval pipelines do not reproduce the safety gains of curated evidence even when they improve accuracy.