Counterfactual Evaluation Reveals Hidden Capability Profiles in Clinical LLMs and Agents

🕒 Published (v1): 2026-05-28 21:37 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Coverage-based metrics (like weighted recall against consensus) cannot distinguish a clinical AI that genuinely responds to changing patient facts from one that outputs the same recommendation regardless of input. This paper introduces the Causal Sensitivity Score (CSS), a pre-registered counterfactual metric that mutates oncology tumor-board cases and scores whether model recommendations update in the correct direction. Applying CSS to six frontier models reveals near-opposite rankings compared to the coverage baseline, plus a universal surgery-status blind spot invisible to coverage metrics.

Problem

Standard clinical LLM evaluation uses coverage-based rubrics (output overlap with reference recommendations), which cannot detect "shortcut" models that always propose the same treatment regardless of patient-specific clinical signals. Two systems can match consensus equally well—one by genuinely reasoning over patient facts, one by memorizing high-prevalence treatments—and existing metrics cannot differentiate them. This failure mode is aggravated in tool-using agents, where a model can retrieve the right information and still ignore it.

Method

The authors define the Causal Sensitivity Score (CSS): a pre-registered interventional metric over 12 mutations across five clinically meaningful families—(A) biomarker flips (HER2/ER/PD-L1 status changes), (B) prior-treatment failure injections, (C) biomarker mention deletions, (D) surgery-status toggles, (E) stage perturbations—applied to 224 oncology tumor-board cases (median ~80k characters, expert-annotated ground truth). For each (model, intervention, case) triple, a baseline and a mutated recommendation are generated; an LLM judge scores {0, 0.5, 1.0} (no change / acknowledged but unchanged / updated correctly) against a pre-registered expected-change direction. CSS is the mean score across applicable tuples. To avoid self-judging, gpt-5.4 is judged by Claude Opus and all others by gpt-5.4; a uniform-Opus cross-replication validates judge independence. The same protocol transfers to a ReAct tool-using agent setting on 100 Family D tuples, where interventions propagate through tool returns (a read_chart_section tool over ten sections) rather than the prompt.

Key Contributions

• Definition and pre-registration of CSS as an interventional complement to coverage-based clinical evaluation, specified before any model is run to prevent post-hoc catalog design.
• Empirical demonstration that CMS and CSS rank six frontier models in nearly opposite orders (Spearman ρ = −0.49; all six change rank), exposing that output quality ≠ input responsiveness.
• Discovery of a universal surgery-status blind spot (Family D ≤ 17.2% CSS for every model from every lab), a clinically meaningful safety gap invisible to CMS.
• Extension of CSS to tool-using ReAct agents with no metric modification, showing tool use lifts CSS for five of six models while one model (gpt-5.4) remains unresponsive despite identical retrieval behavior.
• Cross-judge replication (ρ = +1.00 rank preservation) and three-rater medical-professional validation confirming aggregate population-level findings.
• Framing CSS as a candidate dense reward signal for agentic RL in clinical settings.

Results

• CMS vs. CSS rank reversal: grok-4.20-reasoning ranks 6th on CMS (0.480) but 1st on CSS (0.473); gpt-5 ranks 1st on CMS (0.610) but 4th on CSS (0.427); gpt-5.4 ranks 4th on CMS (0.575) but last on CSS (0.309).
• Family D universal failure: All six models score ≤ 17.2% CSS on surgery-status interventions (grok-4.20: 17.2%, claude-opus-4-7: 11.9%, gpt-5.4: 3.9%). This failure is absent from CMS rankings.
• Per-family profiles: claude-opus-4-7 wins Family A (biomarker flip, 0.523); gpt-5 wins Family C (biomarker strip, 0.406); grok-4.20 wins Family B (prior treatment, 0.868) and Family D (0.172).
• Agent transfer (Family D): Tool use lifts CSS +20.3pp (claude-sonnet-4-6: 0.070→0.273), +17.0pp (gpt-5: 0.065→0.235), +12.0pp (claude-opus-4-7: 0.115→0.235), +6.0pp (grok-4.20: 0.170→0.230), +2.5pp (gpt-5.4-mini: 0.135→0.160); gpt-5.4: 0.050→0.050 (no change). All models call the tool 7.0–8.3 times and query procedures at similar rates, ruling out retrieval differences as an explanation.
• Judge replication: Uniform-Opus re-judging produces ρ = +1.00 rank preservation; inter-judge κ = 0.61–0.69 on {0, 0.5, 1.0}.
• Human validation: LLM-vs-majority κ = 0.46 (69/100 exact agreement); Family D LLM mean CSS = 0.10, human mean = 0.09.

Limitations

• Underpowered rank test: n = 6 models; ρ = −0.49 has exact permutation p = 0.36—rank disagreement is a descriptive pattern, not statistically confirmed.
• Regex-based counterfactuals admit three failure modes scored as 0.0 under the pre-registered rule: semantic no-ops (text changes without clinical-meaning change), incomplete propagation (one section mutated while others still imply the original fact), and medical incoherence (59% of Family D human-annotated tuples flagged as incoherent, e.g., curative resection inserted into a metastatic chart).
• Catalog clinical vetting: Intervention catalog and scoring rules authored by one researcher, not independently reviewed by practicing oncologists prior to submission.
• Scope: Findings specific to oncology tumor-board cases; generalizability to other clinical domains requires domain-appropriate catalog construction.
• Agent attribution: The gpt-5.4 non-response is consistent with structural responsiveness deficit but confounded by sectioned retrieval, ReAct prompting structure, and the 8 KB per-section cap—not a controlled isolation.
• Per-row reliability low for key families: LLM-vs-human κ = 0.16 (Family D) and 0.07 (Family C); headline claims must be read as population-level, not per-case.

Relevance to Foundation Models in Medicine

This paper directly challenges the dominant evaluation paradigm for clinical foundation models by showing that output-coverage metrics systematically misrank models on the clinically critical property of input-responsiveness—a failure mode that becomes more dangerous as LLMs are deployed in high-stakes agentic loops. The universal Family D finding (every frontier model fails to update treatment recommendations when surgery status changes, despite surgery status being directly determinative of treatment timing) is a concrete, pre-registered safety signal for the clinical deployment of current-generation models. For those tracking foundation models in medicine, CSS provides a transferable evaluation primitive for agentic clinical AI that pairs naturally with coverage-based metrics and offers a candidate dense reward for RL fine-tuning, addressing the question of whether clinical foundation models reason causally over patient facts or pattern-match to consensus outputs. The rank-reversal result also implies that leaderboard rankings on existing clinical benchmarks may be misleading proxies for the responsiveness property that matters most in dynamic, real-patient settings.