Skip to content

Tell Me: An LLM-powered Mental Well-being Assistant with RAG, Synthetic Dialogue Generation, and Agentic Planning

🕒 Published (v1): 2025-11-18 12:43 UTC · Source: Arxiv · Venue: ACL · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Tell Me is a three-module mental well-being demo system combining RAG-grounded dialogue, profile-conditioned synthetic client–therapist dialogue generation, and a CrewAI multi-agent planner for adaptive self-care. The paper evaluates only the RAG assistant, benchmarking nine LLMs with LLM-as-a-judge scoring and a 10-person human study. RAG outperforms non-RAG on most dimensions; Claude 3.7 Sonnet ranks highest under GPT-5 judging.

Problem

Three persistent gaps in LLM-based mental well-being systems: (1) context-poor responses yield generic interactions; (2) real therapeutic dialogue data is confidentiality-restricted, limiting reproducibility; and (3) existing well-being tools are static and do not adapt to evolving user emotional states.

Method

RAG assistant: Counsel Chat and Mental Health Counseling Conversations datasets embedded with BAAI/bge-small-en-v1.5 into a LlamaIndex vector store; top-3 retrieved responses re-ranked by a custom therapeutic-tone query engine; multi-turn coherence via a Context Chat Engine memory buffer; sentiment-based safety prefilter for high-risk inputs.

Synthetic dialogue generator: LLM role-plays both client and therapist conditioned on a researcher-supplied client profile (demographics, concerns, history), producing downloadable multi-turn transcripts.

Well-being planner (CrewAI): Three-agent pipeline — transcription analysis agent extracts concerns → Plan Generator Agent produces a 7-day activity/affirmation plan → Meditation Audio Agent synthesizes guided audio via Microsoft Edge Neural TTS.

Evaluation: Nine models (GPT-4o, Claude 3.7 Sonnet, LLaMA-3, Mistral-7B, Gemma-3, Phi-4, three Ollama mental-health variants) scored on 10 scenario prompts by GPT-5 (primary) and GPT-4o (secondary) judges across Safety, Empathy, Usefulness, Clarity, Overall (weighted 3/3/2/2/2). Best model advances to RAG vs. non-RAG human study (n=10, within-subject, 50 judgments, 5-point scale).

Key Contributions

  • Unified open-source demo (Hugging Face Spaces + GitHub) integrating RAG assistant, synthetic dialogue generator, and CrewAI-based agentic planner in a single Streamlit interface.
  • Profile-conditioned synthetic client–therapist dialogue generation to address confidential data scarcity.
  • Multi-LLM benchmark (9 models × 10 prompts × 2 judges = 180 scored outputs) with dual LLM-as-a-judge evaluation for well-being dialogue.
  • Human study confirming RAG grounding improves empathy, clarity, and conversational depth over baseline generation.
  • Sentiment-based safety prefilter integrated into the RAG pipeline for high-risk content interception.

Results

  • GPT-5 judge: Claude 3.7 Sonnet highest (9.5), GPT-4o (8.8), LLaMA-3 (8.6), Gemma-3 (8.5), MentalLLaMA-2 lowest (5.8).
  • GPT-4o judge: GPT-4o self-ranked highest (8.9), Claude and Gemma-3 tied at (8.7), MentalLLaMA-2 lowest (6.5).
  • Human study (RAG vs. non-RAG, n=10): RAG mean 3.8 vs. non-RAG 3.6; RAG better on Clarity (4.2 vs. 3.9) and Helpfulness (4.0 vs. 3.9); non-RAG slightly better on Groundedness (4.0 vs. 3.8).
  • Qualitative: participants described RAG as "organized and on-point" with better validation; non-RAG faster but less engaging.

Limitations

  • Human study: n=10 nonclinical, age 25–35 working professionals only; 50 total judgments; not statistically powered for conclusive claims.
  • Evaluation covers only 10 curated prompts and only the RAG assistant; synthetic dialogue generator and CrewAI planner received no formal quantitative evaluation.
  • LLM-as-a-judge may reflect judge model biases (GPT-4o ranked itself first when used as judge).
  • No licensed clinician review of system outputs or safety prefilter adequacy.
  • Synthetic dialogues cannot fully replicate nuance of real therapeutic exchanges; planner recommendations are not medically validated.
  • System performance depends on underlying LLMs and datasets, which may carry cultural/demographic biases.

Relevance to Agentic AI / LLM Agents

The CrewAI-based Well-being Planner is a concrete example of multi-agent orchestration in a sensitive, user-facing domain: a three-agent pipeline decomposes a complex task (transcript → weekly plan + meditation audio) into specialized roles with tool use (TTS synthesis). The system illustrates agentic planning as an extension beyond single-turn dialogue — translating conversation context into actionable, longitudinally structured outputs. The integration of RAG, synthetic data generation, and agentic planning in one framework makes it a useful reference architecture for researchers building domain-specific LLM agent systems, particularly where data scarcity and safety constraints require modular, controlled pipelines.