Skip to content

Towards Doctor-Like Reasoning: Medical RAG Fusing Knowledge with Patient Analogy through Textual Gradients

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DoctorRAG is a medical RAG framework that emulates clinical reasoning by dually retrieving from both a medical knowledge base and a patient case base, then iteratively refining outputs via Med-TextGrad—a multi-agent textual gradient optimization process. It outperforms Vanilla RAG, Proposition RAG, and Graph RAG across multilingual (Chinese/English/French), multitask (diagnosis, QA, treatment recommendation, text generation) benchmarks.

Problem

Existing medical RAG systems retrieve solely from knowledge bases (clinical guidelines, literature), ignoring case-based experiential knowledge—the analogical reasoning from similar past patients that practicing clinicians routinely apply. Standard dense retrieval also lacks fine-grained concept-level precision for medical queries, and generated outputs lack a dedicated grounding/validation mechanism to ensure faithfulness to retrieved context.

Method

DoctorRAG operates in two stages:

Stage 1 — Expertise-Experience Dual Retrieval: A Declarative Statement Agent converts knowledge chunks into declarative sentences; a Query Tagging Agent annotates both queries and knowledge entries with ICD-10-style concept identifiers. Retrieval from the Knowledge Base uses a concept-constrained cosine similarity score (cosine similarity applied only when concept sets overlap; −∞ otherwise). Retrieval from the Patient Base uses standard cosine similarity over patient complaint embeddings. Top-k results from both sources are concatenated into a unified context.

Stage 2 — Med-TextGrad Iterative Refinement: A multi-agent computation graph models generation as a differentiable forward pass. A Context Criterion agent and a Patient Criterion agent independently evaluate the generated answer, producing textual critiques (the "loss"). Textual gradients are backpropagated: first from critiques to the answer level, then from answer-level gradients to the prompt level. A TGD.step LLM synthesizes both prompt-level gradients to produce a refined prompt for the next iteration (T=3 iterations). Embeddings use OpenAI text-embedding-3-large; preprocessing uses DeepSeek-V3.

Key Contributions

  • Dual-source retrieval combining explicit knowledge (knowledge base) and implicit experiential knowledge (patient case base), with concept-constrained hybrid retrieval
  • Med-TextGrad: a multi-agent textual gradient backpropagation framework for iterative answer refinement, adapted from TextGrad to the medical domain with separate Context and Patient criteria
  • Declarative statement transformation and conceptual tagging pipeline for structured knowledge representation
  • Multilingual (Chinese/English/French), multitask evaluation demonstrating consistent improvements over strong RAG baselines
  • UMAP-validated patient embedding clustering confirming that case-based retrieval captures clinically coherent patient similarity

Results

  • DDXPlus EN disease diagnosis: DoctorRAG (GLM-4-Plus) 98.27% vs. Graph RAG best 92.37% (+5.9 pp)
  • DDXPlus FR disease diagnosis: DoctorRAG (GPT-4.1-mini) 98.73%
  • NEJM-QA (EN nephrology QA): DoctorRAG 72.64% vs. Vanilla RAG 69.58% (+3.1 pp)
  • RJUA text generation Rouge-L: DoctorRAG 31.98 vs. Graph RAG 25.97 (+6.0)
  • DialMed DD: DoctorRAG (GPT-4.1-mini) 94.96%
  • Med-TextGrad pairwise (COD, n=50): T2 preferred over initial answer (OA) in 84% of overall comparisons; all iterations beat ground truth answers in >90% of comparisons
  • Ablation (NEJM-QA): removing Patient Base −1.52 pp (71.73→70.21); removing Knowledge Base −1.21 pp; removing Concept Tagging −1.21 pp
  • BERTScore exception: Vanilla RAG (GPT-4.1-mini) achieves higher BERTScore on RJUA (95.52) and COD (97.04) than DoctorRAG, attributed to DoctorRAG's longer, more comprehensive outputs

Limitations

  • Med-TextGrad adds substantial token cost; performance converges after k≈4 retrieved items, but each additional k linearly increases token consumption
  • Iterative refinement exhibits diminishing returns and potential quality regression beyond T2 (T3 shows no consistent improvement)
  • Patient base construction requires de-identified clinical records that may be unavailable in low-resource settings; 80/20 train/test split means the patient base is dataset-specific
  • BERTScore occasionally favors shorter, more concise baselines—DoctorRAG's verbosity may be undesirable in brevity-constrained clinical contexts
  • Evaluation relies partly on LLM-as-judge (DeepSeek-V3) with only 2 human expert validators on 50 samples, limiting statistical robustness of the qualitative comparison

Relevance to Agentic AI / LLM Agents

DoctorRAG is a concrete instance of multi-agent orchestration for domain-specific reasoning: it decomposes a single query-answering pipeline into specialized LLM agents (declarative statement agent, query tagging agent, context criterion, patient criterion, gradient computation agents, TGD optimizer) that communicate via structured textual signals. The Med-TextGrad module directly operationalizes the agentic pattern of iterative self-refinement through feedback loops, analogous to tool-use-and-reflect cycles in general LLM agent frameworks. For researchers tracking agentic AI, this paper demonstrates how agent-based decomposition can substitute for gradient-based fine-tuning in high-stakes domains, and provides a worked example of textual backpropagation as an alignment/grounding mechanism applicable beyond medicine.