Causal Reinforcement Learning based Agent-Patient Interaction with Clinical Domain Knowledge¶

🕒 Published (v1): 2025-11-17 22:38 UTC · Source: Arxiv · Venue: AAAI · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper introduces Causal Reinforcement Learning (CRL), a Q-learning framework that integrates causal discovery (via LiNGAM/PC algorithms) and counterfactual reasoning into policy optimization for robot-assisted reminiscence therapy with dementia patients. CRL learns a DAG over patient behavioral states and robot actions, using it as an inductive bias to accelerate training. A lightweight deployment mode embeds the learned policy into an LLM system prompt without fine-tuning.

Problem¶

Model-free RL for clinical human-robot interaction suffers from poor sample efficiency, opacity, and an inability to leverage clinical domain knowledge—making it unsafe and impractical for dementia care where data is scarce, exploration must be constrained, and decisions require clinical justification. Existing alternatives are either non-adaptive (rule-based) or data-hungry without interpretability.

Method¶

The framework runs Q-learning over an 18-state MDP (3 response-relevance × 3 emotional-valence × 2 confusion states) with 6 discrete robot actions. A causal DAG is inferred from simulated trajectories using LiNGAM, and Conditional Average Treatment Effects (CATE) are estimated per state cluster via EconML. At each training step, the agent mixes Q-table actions with DAG-suggested actions using configurable weights (CRL-Static: fixed 0.45/0.45; CRL-Dynamic: shifts from DAG-heavy to RL-heavy over epochs). A hard override forces GiveChoice on persistent negative emotion or confusion. Two dementia nurse practitioners validated DAG plausibility. For deployment, the final policy array (length 18) is embedded in a GPT system prompt that performs state inference → policy lookup → action enactment without gradient updates.

Key Contributions¶

First integration of clinical-knowledge-grounded causal discovery into RL for agent-patient interaction in dementia care
CRL-Dynamic adaptive weighting scheme that phases out DAG reliance as training matures
CATE tables quantifying per-state action effects (e.g., Explain yields +4.3 reward under IR/Neutral/Confused)
Lightweight LLM deployment: fixed policy embedded in system prompt, no fine-tuning required
Expert validation of learned DAG by two dementia nurse practitioners

Results¶

CRL-Dynamic achieves superior early cumulative reward vs. model-free RL, CRL-Static, and DAG-Only across 1500 epochs (18-state simulation, 100 test runs)
CRL-Dynamic maintains a higher proportion of high-return episodes (>150 threshold) than DAG-Only and converges faster than pure RL under RL-only execution
Under strategy-consistent evaluation (DAG retained at test time), RL outperforms CRL variants—confirming causal priors help training but hurt execution if retained
CRL-Dynamic reduces low-return episode frequency (<50 threshold) faster than all baselines
Episode length (avg ~44–48 steps): CRL variants match RL speed of convergence; DAG-Only plateaus lower
CATE example: Comfort under NR/Neg/Confused = +2.3 reward vs. EasyPrompt baseline
LLM (GPT-5) enacts 5 of 6 reachable policy actions correctly in qualitative trace

Limitations¶

Validated entirely in simulation; no real patient or robot experiments
18-state space is coarse—3 dimensions with discrete categories may miss clinically relevant nuance
DAG learned from simulated (not real) interaction data; causal structure may not transfer to clinical reality
LLM deployment evaluation is qualitative only (one example trace)
Causal discovery algorithms (LiNGAM, PC) assume specific noise/linearity conditions that may not hold
Expert validation feedback on DAG refinement was not incorporated into the model

Relevance to Agentic AI / LLM Agents¶

This work demonstrates a hybrid architecture where a structured RL policy (learned with causal priors) controls an LLM agent via system-prompt injection—a concrete pattern for constraining LLM behavior to verified, interpretable decision policies without fine-tuning. The CRL-Dynamic weighting mechanism is a form of curriculum-guided exploration relevant to any agentic system operating in high-stakes, low-data environments. The CATE-grounded action selection is directly applicable to reward-shaping in LLM agent pipelines where actions have heterogeneous effects across user states. The simulation-to-LLM deployment pipeline offers a blueprint for grounding conversational agents in clinically validated behavioral models.