Skip to content

Knowledge-augmented Agentic AI for Mental Health Medication Information Seeking

🕒 Published (v1): 2026-06-24 16:54 UTC · Source: Arxiv · link

Why this paper was selected

Multi-source knowledge-augmented agent for drug safety; concrete harness integrating regulatory + patient-narrative retrieval

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

A provenance-aware, multi-agent framework integrates 466,525 Reddit posts, 60,782 WebMD reviews, and 20 years of FAERS records for nine antidepressants into a Neo4j knowledge graph, then exposes it through a six-agent chatbot pipeline for psychiatric medication education. Patient-generated community data form a partly independent pharmacovigilance signal, often preceding FDA adverse-event dates by hundreds of days. The system is designed as an educational aid, not a clinical decision tool.

Problem

Psychiatric medication safety knowledge is split between authoritative-but-abstract regulatory records (FAERS) and experience-near-but-unvalidated patient narratives (Reddit, WebMD). Current LLMs cannot reliably integrate these streams without hallucination or evidence conflation—especially dangerous in psychiatry, where poorly framed adverse-event information can amplify nocebo responses and treatment non-adherence.

Method

Data pipeline. An ATC-N drug dictionary (626 generic names + LLM-expanded brand synonyms) drove keyword-based collection. Rule-based and LLM-based filtering yielded 466,525 Reddit posts and 60,782 WebMD reviews; FAERS quarterly dumps were parsed into SQLite and aligned to the same 2005–2025 window.

NER. Nine LLMs were benchmarked against a physician-annotated gold standard covering medications (with dosage, form, duration, continuation), primary/comorbid conditions, and side effects (with severity, frequency, duration). GPT-4.1-mini was selected as the production extractor based on accuracy/cost/throughput trade-off.

Entity canonicalization. Raw NER strings were mapped to ATC-N (medications), ICD-10 (conditions), and MedDRA Preferred Terms (side effects) via embedding-based nearest-neighbour retrieval (text-embedding-3-small) with per-entity-type cosine-similarity thresholds calibrated to maximize Youden's J on an annotated gold standard.

Knowledge graph. Implemented in Neo4j with node types {Post, Medication, Condition, SideEffect} and typed edges: TREATS, CAUSES, CAUSES_BY_WITHDRAW, COMORBID_WITH, and MENTIONS (post→entity provenance). Full post text is offloaded to a linked SQLite sidecar with full-text indexing, keeping the graph compact.

Multi-agent chatbot. A six-stage pipeline processes each user query: 1. NER Agent — extracts and canonicalizes medication entities 2. User Intent Agent — classifies query type (general AE, demographic-stratified, temporal) 3. KG Query Generation Agent — selects source graphs (FAERS/WebMD for structured demographics; Reddit for experiential context) and generates Cypher queries 4. Summarization Agent — produces independent, provenance-tagged per-source summaries 5. Comparison Agent — synthesizes cross-source consensus and divergence 6. Validation Agent — screens against a safety ruleset before returning the response

Key Contributions

  • Scalable LLM-based NER pipeline benchmarked across nine frontier models on physician-annotated psychiatric medication data
  • Multi-source comparative analysis (AE overlap, frequency rank, source balance, temporal lead-time) across FDA, WebMD, and Reddit for nine antidepressants
  • Neo4j knowledge graph grounded in ATC-N/ICD-10/MedDRA with MENTIONS-edge provenance for evidence-traced retrieval
  • Six-agent chatbot harness with intent-driven source routing and a terminal Validation Agent safety gate

Results

  • NER (name-level F1): medication — GPT-4.1-mini 0.969; condition — Claude-Sonnet-4 0.973; side effect — Deepseek-V3 0.912
  • Attribute-level extraction is substantially weaker: dosage F1 0.523–0.751; side-effect duration F1 0.188–0.476; side-effect frequency F1 0.125–0.750
  • Community source agreement: WebMD–Reddit Jaccard up to 0.905 (desvenlafaxine), 0.875 (duloxetine), 0.786 (sertraline)
  • Regulatory vs. community overlap is lower: FDA–Reddit Jaccard = 0.558, FDA–WebMD = 0.577 for sertraline
  • Frequency correlation: Reddit–WebMD r = 0.847 vs. FDA–Reddit r = 0.593, FDA–WebMD r = 0.588 for sertraline
  • For sertraline, many adverse events appeared in community sources hundreds of days before the corresponding FDA receipt date
  • Source-balance entropy varies by drug: fluoxetine/sertraline/desvenlafaxine show balanced cross-source AE distribution; duloxetine/amitriptyline are more concentrated

Limitations

  • English-only text; cross-lingual generalizability untested
  • Prospective evaluation of chatbot usefulness and patient benefit not yet conducted
  • Attribute-level NER (dosage, duration, frequency) remains weak across all nine models, limiting structured pharmacokinetic reasoning
  • Community data are self-reported and heterogeneous in accuracy and representativeness; results described as requiring further validation
  • Coverage limited to nine antidepressants; ATC-N scope not fully exercised
  • Temporal lead-time analysis assumes FDA receipt date as the regulatory benchmark, which conflates reporting lag with true signal emergence

Relevance to Harnesses / Meta-Harnesses

This paper is a concrete instantiation of a domain-specific agent harness: a fixed pipeline of six specialist agents with explicit routing logic (intent-driven source selection) and a terminal Validation Agent acting as a safety gate before output—a pattern directly analogous to meta-harness designs that decompose complex tasks into ordered specialist sub-tasks. The provenance-preserving knowledge graph functions as the shared state substrate (analogous to a harness context store), ensuring each downstream agent can trace claims to their source without re-querying raw corpora. The intent-driven conditional routing—FAERS/WebMD for structured demographic queries, Reddit for experiential context—demonstrates how harness-level orchestration logic can encode domain knowledge about tool selection rather than delegating that choice to the LLM at inference time. For researchers tracking harness design, this paper provides a safety-critical deployment case study showing how provenance, source-separation, and a validation gate interact within an agentic pipeline.