Doctor-R1: Mastering Clinical Inquiry with Experiential Agentic Reinforcement Learning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Agentic RL for clinical inquiry; models doctor-patient consultation strategically

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Doctor-R1 is an 8B-parameter clinical AI agent trained via Experiential Agentic Reinforcement Learning (RL) to master dynamic multi-turn patient inquiry alongside diagnostic decision-making. It frames outpatient consultation as a POMDP and optimizes the policy with GRPO using a two-tiered reward covering both conversational soft skills and diagnostic hard skills. An 8B Doctor-R1 surpasses GPT-4.1, Grok-4, and Claude Sonnet 4 on HealthBench despite having far fewer parameters.

Problem

Frontier LLMs achieve above-human scores on static benchmarks (USMLE, MMLU-Med) but fail at dynamic clinical inquiry — the real-world process of sequentially gathering evidence through targeted questioning, performing differential diagnosis on-the-fly, and communicating with empathy under uncertainty. Existing medical agents lack: (1) strategic, adaptive multi-turn questioning, (2) empathetic communication, and (3) a principled mechanism to learn from high-quality prior consultation trajectories.

Method

Doctor-R1 trains a doctor agent \(\pi_\theta\) (initialized from a base 8B LLM) within a closed-loop multi-agent POMDP \(\langle S, A, O, R \rangle\):

• Dynamic Interactive Environment: A Patient Agent (LLM initialized with hidden clinical scenarios) simulates patient responses, producing observations \(o_t\) (dialogue history). The Doctor Agent acts by generating questions or a diagnosis.
• Two-Tiered Reward Architecture:
• Process reward \(R_\text{turn}\): evaluates each utterance across 8 dimensions (Safety, Reasoning, Medical Accuracy, Completeness, Information Gathering, Faithfulness, Empathy/Clarity, Humility) with a hierarchical veto system — safety/reasoning/accuracy failures trigger immediate penalties (\(-1.0\) or \(-0.75\)) overriding the weighted sum.
• Outcome reward \(R_\text{final} \in \{0.0, 0.5, 1.0\}\): terminal reward for diagnostic correctness.
• Experience Repository: Stores high-reward tuples \((s_t, a_t, R_t)\) and retrieves them via a 3-stage pipeline: (1) embedding-based candidate selection with combined score \(S_\text{combined}(Q, E_i) = f_\text{sim}(f_\text{emb}(Q), f_\text{emb}(E_i^{\text{state}})) + \alpha \cdot R(E_i)\); (2) cross-encoder reranking for high-fidelity relevance; (3) novelty and dynamic reward filtering using threshold \(\tau_\text{dynamic} = \mu_R + \beta_\text{std} \cdot \sigma_R\). Retrieved top-\(k\) experiences are prepended to the agent's context as in-context guidance.
• Policy optimization uses GRPO (Group Relative Policy Optimization), contrasting a high-reward action \(y_c\) against a group of rejected alternatives via: $\(\mathcal{L}_\text{GRPO} = -\mathbb{E} \left[ \log \frac{e^{R_\psi(x, y_c)}}{\sum_{y \in \{y_c\} \cup \{y_r\}} e^{R_\psi(x, y)}} \right]\)$

Key Contributions

• First agent framework unifying strategic multi-turn inquiry (soft skills) and medical decision-making (hard skills) within a single 8B model.
• Experiential Agentic RL methodology: closed-loop multi-agent training environment + two-tiered hierarchical reward + experience repository with multi-stage retrieval.
• Empirical demonstration that enhancing inquiry capability causally improves diagnostic accuracy — not just communication scores.
• 8B model outperforming 32B open-source models and frontier proprietary LLMs on clinical consultation benchmarks.

Results

• HealthBench Main (Avg. Score): Doctor-R1 (36.29) > Grok-4 (33.03) > GPT-4.1 (31.18) > Claude Sonnet 4 (25.69); +9.91 over UltraMedical-8B (26.38).
• HealthBench vs. open-source: Surpasses Baichuan-M2-32B (36.29 vs. 33.16, +9.4%) using 4× fewer parameters; outperforms it on Accuracy (37.84 vs. 33.95), Communication Quality (64.15 vs. 58.01), Context Awareness (49.24 vs. 46.80).
• MAQuE: Doctor-R1 (60.00) vs. UltraMedical-8B (52.00), +8.00; matches GPT-4.1 in Accuracy while achieving far superior Empathy scores (93.80 vs. 75.20 vs. GPT-4.1).
• Communication vs. Baichuan-M2: +25.05 on HealthBench communication scores (47.16 vs. 22.11).
• Static benchmarks (MedQA, MMLU): Performance maintained, confirming training does not degrade foundational medical knowledge.
• Human expert evaluations confirm superior clinical capability and patient-centric performance.

Limitations

• Patient Agent and Consultation Evaluator are themselves LLMs, introducing potential reward hacking or systematic evaluation bias not corrected by real patient data.
• Experience repository at scale (millions of entries) may introduce retrieval latency or stale-knowledge issues as the distribution of training scenarios evolves.
• Evaluation primarily on English-language benchmarks (HealthBench, MAQuE); cross-lingual and cross-cultural clinical consultation generalization is untested.
• The two-tiered reward weights and veto thresholds (\(\epsilon\), \(\beta_\text{std}\), \(\tau_\text{reward}\)) are manually configured hyperparameters with no ablation of sensitivity reported in the provided text.
• Safety-critical deployment requires real clinical trials; benchmark superiority over proprietary models does not directly imply clinical deployment readiness.

Relevance to Agentic AI / LLM Agents

Doctor-R1 is a concrete instantiation of the emerging Agentic RL paradigm — training LLMs as goal-directed sequential decision-makers in partially observable environments rather than as supervised imitators. Its experience repository with multi-stage retrieval is a practical mechanism for in-context experience replay, directly relevant to long-horizon agent training where full fine-tuning after each episode is impractical. The two-tiered process+outcome reward with hierarchical safety veto addresses a general challenge in multi-objective agentic reward shaping, and the GRPO-based optimization within a multi-agent simulation loop is a transferable template for other interactive agent domains (e.g., negotiation, tutoring, customer service). The finding that inquiry capability improvements transfer to downstream decision accuracy has implications for the design of information-seeking agents more broadly.