The strength of clinical evidence is recoverable from language model representations but not from their stated grades¶

🕒 Published (v1): 2026-06-27 18:06 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

LLMs internally encode an ordered signal of clinical evidence strength (recoverable via linear probing, median AUROC 71.8) yet completely fail to verbalize that signal when prompted (verbalized accuracy falls to chance). This dissociation—25–27 percentage points between what is encoded and what is stated—holds across all 22 tested open-weight models spanning general, medical, and reasoning families.

Problem¶

Clinical practice requires knowing not just whether an LLM's assertion is correct but how strongly it is supported by evidence. LLMs communicate confidence poorly, and it is untested whether they internally register evidence strength (a graded property distinct from factual truth) and whether any such internal signal is genuine or merely a surface/lexical artifact.

Method¶

The authors compiled 45,134 atomic clinical claims from six public biomedical sources (USPSTF, CPSTF, medical reversals, Trialstreamer, Evidence Inference, EBM-NLP). 20,611 claims were harmonized to a four-level ordinal evidence grade (high/moderate/low/very low) under three independent frameworks. For each of 22 frozen open-weight LLMs (0.6–70B), the residual stream activation at the final claim token was extracted per transformer layer. An L2-regularized multinomial logistic probe was trained on these hidden states to decode the grade. Verbalized grades were elicited via zero-shot and few-shot prompting with a fixed four-choice format. Controls included: (1) TF-IDF/surface-covariate baselines; (2) within-source fitting to hold writing style constant; (3) partial Spearman correlation after residualizing out lexical scores; (4) a matched-text test to remove lexical advantage; (5) a leave-one-framework-out transfer test; and (6) a balanced grade-by-truth design using "crossing cells" (e.g., reversed-high claims: strong grade but false) to decorrelate grade from truth.

Key Contributions¶

First systematic test of whether clinical evidence grade is linearly decodable from LLM hidden states across a large, multi-source, multi-framework corpus.
Demonstration of a consistent and large internal-versus-verbalized dissociation: probes achieve median AUROC 71.8 while verbalized accuracy is near-chance across all 22 models.
Evidence that the recoverable signal is largely lexical and does not transfer across clinical topics or grading frameworks (leave-one-framework-out results substantiate this).
Demonstration that the hidden evidence-grade signal is geometrically separable from factual truth using crossing-cell controls.
External-probe flagging of weakly supported claims (AUROC 69.2) as a potential safety application despite models' inability to self-report grades.

Results¶

Linear probe decodability: Median macro one-versus-rest AUROC = 71.8 across all 22 models; above-chance in every model.
Verbalized accuracy: Stated grades fall to chance; the behavioral gap is 25–27 percentage points below the probe (McNemar's test confirms systematic discordance).
Scale: Decodability does not increase with parameter count; OLS over the Qwen3 dense ladder (0.6–32B) shows no significant trend.
Reasoning models: Decodability is weakest in reasoning-distilled models despite (or because of) their chain-of-thought training.
Medical domain-adapted models: No consistent advantage over matched general-domain base models.
Weak-claim flagging: Probe AUROC for identifying very-low-evidence claims = 69.2, above chance despite no verbalization signal.
Lexical confound: TF-IDF baseline alone recovers meaningful signal; partial Spearman after residualizing lexical scores attenuates but does not eliminate the hidden-state advantage in within-source analyses.
Cross-framework transfer: Leave-one-framework-out performance drops substantially, confirming the signal does not generalize across grading rubrics.

Limitations¶

The recovered signal is largely lexical and tied to surface properties of claim text; it does not robustly transfer across clinical topics or grading frameworks.
The corpus is heavily imbalanced by grade (10,002 low vs. 513 high) and confounded by source and writing style, mitigated but not fully eliminated by design controls.
USPSTF is the only source spanning the full grade range within a single rubric; crossing-cell analysis relies on small \(n\) (e.g., weak-true: \(n=7\), settled-false: \(n=19\)), requiring bootstrap CIs.
Six of 22 models were excluded from the verbalized analysis for producing degenerate or unparseable outputs; the behavioral gap estimate is conservative but based on the remaining 16.
Larger models (≥27B) were loaded in 4-bit quantization due to memory constraints, potentially affecting representation quality.
Only frozen, instruction-tuned open-weight models were tested; closed-source models (GPT-4, Claude) and fine-tuning regimes are not covered.
The study does not test whether prompt engineering or chain-of-thought elicitation could close the gap.

Relevance to Foundation Models in Medicine¶

This paper directly challenges the assumption that medical LLMs—including domain-adapted and reasoning-distilled variants—reliably communicate the epistemic weight behind their clinical assertions, a core safety requirement for clinical deployment. It adds a mechanistic dimension to the broader calibration literature: the information about evidence strength exists inside these models but is suppressed or disconnected from their output layer, suggesting that RLHF or instruction-tuning may actively degrade uncertainty expression even as it improves fluency. For researchers tracking foundation models in medicine, the finding that external linear probes can flag weakly supported claims (AUROC 69.2) points toward interpretability-based post-hoc safety wrappers as a near-term mitigation, while the failure of medical specialization (BioMistral, Meditron, Med42, MedGemma) to improve verbalization underscores that domain adaptation alone does not solve clinical trustworthiness.