Skip to content

Tell Me: An LLM-powered Mental Well-being Assistant with RAG, Synthetic Dialogue Generation, and Agentic Planning

🕒 Published (v1): 2025-11-18 12:43 UTC · Source: Arxiv · Venue: ACL · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Tell Me is a three-module mental well-being demo system combining a RAG-based reflective chatbot, a synthetic client–therapist dialogue generator conditioned on user profiles, and a CrewAI-orchestrated planner that generates weekly self-care plans and guided meditation audio. The RAG assistant is evaluated across nine LLMs using an LLM-as-a-judge framework and a small human study. The paper positions the system as a lightweight, open-source testbed for responsible LLM applications in mental health.

Problem

Three persistent gaps in LLM-based mental well-being systems: (1) responses are generic due to insufficient context; (2) real therapeutic dialogue data is scarce due to confidentiality restrictions, limiting dataset availability; (3) most well-being tools are static and do not adapt to individual needs or evolving emotional states.

Method

Three independent modules unified under a Streamlit front-end:

  1. RAG assistant: Embeds two open-source counseling datasets (Counsel Chat, Mental Health Counseling Conversations) with BAAI/bge-small-en-v1.5 into a LlamaIndex vector store; retrieves top-3 passages re-ranked by a custom query engine; maintains conversation history via a Context Chat Engine memory buffer; routes high-risk inputs to a sentiment-based safety prefilter.

  2. Synthetic dialogue generator: Given a researcher-defined client profile (demographics, concerns, history), an LLM role-plays both client and therapist to produce multi-turn transcripts for data augmentation and training.

  3. CrewAI planner: A three-agent pipeline — (i) transcription analysis agent extracts concerns, (ii) Plan Generator Agent produces a seven-day activity/affirmation schedule, (iii) Meditation Audio Agent synthesizes guided meditation via Microsoft Edge Neural TTS.

Evaluation uses GPT-5 and GPT-4o as LLM judges scoring 9 models × 10 prompts on Safety, Empathy, Usefulness, Clarity, and Overall (weighted 3/3/2/2/2), plus a within-subject human study (n=10) comparing RAG vs. non-RAG on 5-point Likert scales.

Key Contributions

  • Unified open demo integrating RAG dialogue, profile-conditioned synthetic dialogue generation, and CrewAI agentic self-care planning in a single Streamlit interface.
  • Multi-LLM benchmark (9 models, 10 prompts, dual judges) for mental well-being dialogue quality.
  • Human study (n=10, 50 judgments) comparing RAG vs. non-RAG outputs on helpfulness, supportiveness, clarity, groundedness, and overall quality.
  • Open-source release (GitHub + Hugging Face Spaces) with user-supplied API keys for privacy and reproducibility.
  • Demonstration that a CrewAI three-agent pipeline can translate conversation into personalized weekly plans and TTS meditation audio.

Results

  • LLM-as-a-judge (GPT-5): Claude 3.7 Sonnet ranked 1st (9.5), GPT-4o 2nd (8.8), LLaMA-3 3rd (8.6); MentalLLaMA-2 lowest (5.8).
  • LLM-as-a-judge (GPT-4o): GPT-4o ranked itself 1st (8.9); Claude and Gemma-3 tied 2nd (8.7 each); MentalLLaMA-2 lowest (6.5).
  • Human study: RAG assistant mean rating 3.8 vs. non-RAG 3.6; RAG superior on Clarity (4.2 vs. 3.9) and Helpfulness (4.0 vs. 3.9); non-RAG slightly higher on Groundedness (4.0 vs. 3.8).
  • Qualitative: RAG described as "organized and on-point"; non-RAG faster but less engaging.

Limitations

  • Human study is small (n=10, nonclinical, ages 25–35) and findings are indicative, not conclusive.
  • Only the RAG assistant was formally evaluated; synthetic dialogue generator and CrewAI planner received no quantitative evaluation.
  • LLM-as-a-judge may reflect judge model biases (GPT-5 favors relational depth; GPT-4o favors actionable scaffolding).
  • Evaluation covers only 10 curated prompts; no licensed clinicians reviewed the system.
  • Planner-generated routines are not medically validated.
  • System is not a clinical tool; safety prefilter and disclaimers are heuristic, not clinically certified.
  • Underlying datasets and LLMs may embed cultural/demographic biases.

Relevance to Harnesses / Meta-Harnesses

Tell Me's CrewAI-based Well-being Planner is a direct example of a domain-specific agentic harness: three specialized agents (extraction → planning → TTS synthesis) are orchestrated in a fixed pipeline to transform a conversation transcript into structured outputs. This exemplifies the micro-harness pattern — lightweight multi-agent coordination with defined roles and sequential information flow — that meta-harness researchers study when characterizing how orchestration frameworks (CrewAI, LangGraph, etc.) are instantiated in vertical domains. The paper also demonstrates a two-track evaluation harness (LLM-as-a-judge + human study) that itself functions as a reusable assessment scaffold for dialogue quality, relevant to anyone building evaluation meta-harnesses for agentic systems. Its open-source, modular architecture makes it a concrete case study for how domain harnesses expose interfaces between retrieval, generation, and planning sub-agents.