From Medical Records to Diagnostic Dialogues: A Clinical-Grounded Approach and Dataset for Psychiatric Comorbidity

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper constructs PsyCoTalk, a dataset of 3,000 multi-turn psychiatric diagnostic dialogues covering comorbid conditions, generated via a three-agent pipeline (doctor, patient, tool) grounded in a Hierarchical Diagnostic State Machine (HDSM) and Diagnostic Context Tree (DCT) derived from the SCID-5-RV clinical interview protocol. The pipeline first synthesizes 502 patient EMRs from real social-media posts, then generates diverse fictitious experiences per EMR to drive the agents. Psychiatrists validated dialogue realism and diagnostic correctness.

Problem

Existing mental-health dialogue corpora treat each psychiatric disorder in isolation and lack structured annotation for symptom co-occurrence. No prior large-scale dataset or simulator supports multi-disorder (comorbidity) diagnostic reasoning grounded in DSM-5 standards, leaving LLMs unsystematically evaluated on multi-disorder screening tasks.

Method

Stage 1 — EMR synthesis. Starting from the PsySym Reddit corpus (5,624 users with symptom annotations), users are filtered by symptom diversity (\(\geq\)20 distinct symptom types, \(\geq\)10 symptom posts). A modular pipeline extracts each EMR section independently—dual classifiers (symptoms + life events) for chief complaint, keyword search for demographics, LLM inference for history sections—and merges them. GPT-4o-mini generates the text; eight psychiatrists review iteratively. Output: 502 EMRs across six comorbidity combinations of MDD, AD, BD, ADHD.

Stage 2 — Experience augmentation. Each EMR yields 5 personal histories (\(D_\text{his}\)) and 10 fictitious experiences (\(D_\text{fic}\)) via prompt-based LLM generation: $\(\tilde{e} = \text{LLM}_\text{Prompt}(h, e), \quad h \in D_\text{his},\ e \in D_\text{fic}\)$ enabling up to 50 unique fictitious experience descriptions (FEDs) per EMR.

Stage 3 — Multi-agent dialogue generation. A tool agent orchestrates a doctor agent and patient agent using two symbolic structures: - HDSM: four disorder-specific sub-state machines (130+ states total), each derived from SCID-5-RV with three levels—High-Level States (HLS), Intermediate-Level States (ILS), and Basic-Level States (BLS). Symptom responses are binarized (present/absent); state transitions are rule-gated by response counts. - DCT: a tree controller managing Family History, Personal History, and Experience Inquiry branches, dynamically triggered per turn.

The tool agent's key functions include ResponseClassifier, SSM_OrderGen, NeedExpBranch, BuildPrompt, and IsDialEnd. All three agents run on Qwen2.5-72B. Per EMR, 2 scheduling strategies (symptom-informed vs. random order) × 5 FEDs × 1 doctor profile each → ~5,000 dialogues; filtered to 3,000 validated ones.

Key Contributions

• 502 synthetic clinically grounded EMRs (PsyCoProfile) covering six comorbidity combinations of MDD, AD, BD, ADHD, validated against ~1,000 real de-identified records.
• HDSM + DCT clinical interview pipeline formalizing SCID-5-RV into 130+ symbolic diagnostic states, with binary symptom thresholding for stable LLM-agent transitions.
• PsyCoTalk: 3,000 multi-turn diagnostic dialogues—the first large-scale comorbidity-focused psychiatric dialogue dataset—validated by licensed psychiatrists for realism and diagnostic validity.
• Disease–Symptom Description Knowledge Graph (DSD-KG) used by the patient agent to suppress hallucinated symptom affirmations.

Results

• PsyCoTalk contains 3,000 multi-turn dialogues derived from 502 EMRs; dialogues are described as longer and clinically deeper than existing corpora (e.g., MDD-5k's 5,000 single-disorder simulations).
• Psychiatrist evaluation confirms sufficient diversity and clinical plausibility; structural/linguistic fidelity (dialogue length, token distribution, diagnostic reasoning strategies) judged high vs. real clinical transcripts.
• Synthetic EMR distributions broadly align with real-world data across disease prevalence, age, gender, and family history (Figure 3); age skews younger (peak 20–24 vs. 30–34 in real records), attributed to social-media demographics.
• (No numeric accuracy/F1/BLEU benchmark numbers are reported in the provided text.)

Limitations

• No downstream LLM fine-tuning or evaluation results are reported in the provided excerpt; dataset utility for comorbidity screening models is asserted but not demonstrated quantitatively.
• Binary symptom binarization discards sub-threshold severity distinctions, a known clinical simplification acknowledged by the authors.
• EMR age distribution skews younger than real clinical records due to Reddit demographics.
• Mental status examination and auxiliary test fields are excluded from EMRs due to social-media data limitations.
• All agents use a single model (Qwen2.5-72B); cross-model generalizability of the pipeline is untested.

Relevance to Harnesses / Meta-Harnesses

PsyCoTalk's tool agent is a textbook meta-harness: it sits above the doctor and patient agents and drives them via symbolic state machines (HDSM) and a context tree (DCT), dynamically dispatching topics, classifying responses, managing branching, and deciding termination—decoupling orchestration logic from the generative LLMs entirely. The pattern of formalizing domain expertise as a symbolic controller that governs LLM agent behavior (rather than letting LLMs free-generate) is a recurring design principle in clinical and safety-critical harnesses. The modular EMR generation pipeline (Stage 1) similarly functions as a data harness—chaining classifiers, keyword extractors, and LLM inference steps with expert-review loops—demonstrating how harnesses enable auditable, multi-step synthetic data construction at scale.