Skip to content

MedDDC-Eval: Diagnosis-Decoupled Evaluation of Multi-Turn Medical Consultation Agents

🕒 Published (v1): 2026-07-21 11:32 UTC · Source: Arxiv · link

Why this paper was selected

Decouples inquiry policy from diagnosis quality in multi-turn medical consultation agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Multi-turn medical consultation agents are typically evaluated end-to-end, but this conflates the quality of the agent's evidence-acquisition policy with its terminal diagnosis generation. MedDDC-Eval decouples these by routing every agent's elicited history through a single frozen diagnostic reader (DeepSeek-v3), making diagnosis scores attributable to the history rather than the generator. GRPO-trained Qwen3-32B improves over its initialization by 9.7 and 4.6 total-score points on two held-out splits when optimized with the proposed signals.

Problem

Existing multi-turn medical consultation benchmarks let the same agent both elicit patient history and produce the terminal diagnosis, creating an attribution confound: a strong diagnosis generator can compensate for a thin history, while a weak generator can obscure a rich one. Under coupled evaluation, diagnosis F1 jointly reflects history usefulness and agent-specific generation; replacing only the diagnostic reader on fixed histories shifts F1 by 2.2–19.0 pp and reverses 18–36% of pairwise policy orderings, demonstrating the confound is material.

Method

MedDDC-Eval holds terminal diagnosis generation constant by applying a single frozen DeepSeek-v3 prompt/decoding interface (the "shared reader") to every completed consultation history \(h_\pi\). The primary comparison object is thus the policy-elicited history, not the end-to-end diagnosis output.

Data: Two held-out Chinese-language splits — 100 cases from de-identified hospital records (Record) and 70 cases from online multi-turn consultations (Dialogue); these are case-disjoint from the 1,824 RL training prompt rows. A companion grounded patient simulator responds with record- or queue-grounded facts; the doctor receives only the live interaction.

D/T/E harness: An LLM-assisted judge proposes directional, protocol-admissible semantic coverage edges \(C \subseteq P \times R\) between predicted units \(P\) and reference units \(R\). A deterministic program then computes the maximum-cardinality bipartite matching \(M \subseteq C\), yielding: $\(\text{TP} = |M|,\quad \text{FP} = m - |M|,\quad \text{FN} = n - |M|\)$ Three axes are reported: Diagnosis F1 (\(D\)), Trajectory F1 (\(T\), coverage of expert-annotated clinical information needs), and Efficiency (\(E\), timing score penalizing late or ineffective elicitation). The composite: $\(\text{Total} = 0.5D + 0.4T + 0.1E\)$

Policy optimization: Standard GRPO over interactive multi-turn rollouts using the SLIME framework. Complete doctor–patient trajectories (up to 7 turns) are rolled out; trajectory-level credit assignment broadcasts normalized sequence rewards. The global reward combines diagnosis-result and trajectory-F1 primary signals with auxiliary behavioral constraints (turn budget, format, repetition, question count, prefix similarity).

Key Contributions

  • MedDDC-Eval testbed: 170 held-out cases across two source strata with a frozen shared reader that decouples between-agent diagnosis generation from elicited-history quality.
  • Fixed-history intervention: Empirically demonstrates that changing only reader identity shifts rankings, motivating the shared-reader control.
  • D/T/E protocol: Diagnosis and trajectory F1 as non-substitutable axes; directional one-to-one semantic matching prevents double-counting; efficiency as an auxiliary timing signal; Total as a pre-specified weighted composite.
  • Evaluation-guided GRPO: Shows MedDDC-Eval signals can be used as rewards to improve evidence-acquisition policy; reward ablations quantify contributions of both primary signals.

Results

  • Reader replacement on fixed histories shifts diagnosis F1 by 2.2–15.8 pp (Record) and 3.8–19.0 pp (Dialogue); reverses 5/28 (18%) Record and 10/28 (36%) Dialogue pairwise policy orderings.
  • Trained Qwen3-32B + GRPO improves over its Qwen3-32B initialization by +9.7 total-score points (Record) and +4.6 total-score points (Dialogue) under the shared evaluation reader.
  • Ablation removing the diagnosis-result reward lowers held-out joint performance; ablation removing trajectory F1 similarly degrades performance — both primary reward signals contribute.
  • External baselines include GPT-5.4, GLM-5, HuatuoGPT-o1-70B, Doctor-R1, Baichuan-M2-32B, and Llama-3-70B-UltraMedical; rankings differ substantially between policy-bound and shared readers.

Limitations

  • Reader-conditional scores: Diagnosis F1 remains conditional on the shared reader's capability; a different frozen reader would yield different absolute scores and could affect relative rankings.
  • Language scope: Both source strata are Chinese-language; generalizability to other languages and healthcare systems is untested.
  • Small held-out sets: 100 and 70 cases respectively; statistical power for fine-grained policy comparisons is limited.
  • Simulated patients only: Grounded simulator approximates real patient behavior but does not capture actual patient variability, emotional states, or voluntary disclosure patterns.
  • Data release constraints: Patient-derived text and fine-grained records are withheld; only schema, prompts, scoring code, aggregate results, and synthetic examples are unconditionally released.
  • Single training family: The controlled policy-optimization study uses only Qwen3-32B; same-family comparisons do not validate cross-architecture generalization of the training signals.

Relevance to Agentic AI / LLM Agents

MedDDC-Eval directly addresses how to evaluate and train LLM agents that must actively acquire information over multiple turns rather than reason over pre-supplied context — a defining property of agentic systems. The attribution confound it identifies (policy quality entangled with generation quality) is general to any sequential-decision agent evaluated on terminal outcomes, making the shared-reader control principle applicable beyond medicine. The GRPO training loop over interactive multi-turn rollouts with trajectory-level credit assignment is a clean instantiation of process-and-outcome reward design for agent policy optimization, complementing work like Doctor-R1 and broader multi-turn RL for LLMs. For researchers tracking agentic benchmarking methodology, the one-to-one semantic matching protocol offers a reproducible alternative to unconstrained LLM-judge scoring for open-ended agent outputs.