Dementia-Agents: A Multi-Modal Multi-Agent System for Dementia Staging and Phenotyping

🕒 Published (v1): 2026-06-19 07:16 UTC · Source: Arxiv · link

Why this paper was selected

Multi-modal multi-agent orchestration for clinical staging; strong medical-AI+agent harness fit

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Dementia-Agents is a clinically aligned multi-agent framework that decomposes dementia assessment into five domain-specific expert agents (risk, cognitive, collateral, linguistic, visuographic) whose probabilistic outputs are aggregated by a coordinator agent. Evaluated on a private real-world cohort of 1,066 patients, it outperforms monolithic MLLMs and prior medical multi-agent systems on both three-stage dementia staging and 12-class phenotyping. The framework explicitly handles heterogeneous, partially observed clinical data — a property absent from most prior AI dementia work.

Problem

Most AI dementia work frames diagnosis as binary AD detection or three-stage AD progression on curated research cohorts with complete modality availability. Real-world dementia assessment involves syndrome-level heterogeneity (12+ phenotypes, multiple etiologies), multi-party informants (patients, caregivers, multiple clinicians), and routinely missing data — none of which are modeled by existing medical multi-agent systems, which also lack dementia-specific expert decomposition.

Method

Three-step pipeline:

1. Data Agent: Converts tabular clinical records (demographics, cognitive test results, caregiver questionnaires, drawings, speech transcripts) into structured textual representations that preserve missing-value semantics and routes inputs to five domain-aligned expert agents.

2. Expert Agents (fine-tuned independently): Each expert \(s\) uses a pretrained encoder \(\mathcal{E}_s(\cdot; \theta_s)\) (Qwen3-VL-Embedding-8B for multi-modal; Qwen3-Embedding-8B for text-only) with LoRA adapters, mean-pooled to \(h_s\), then projected via a two-layer MLP \(f_s(\cdot; \phi_s)\): $\(h_s = \text{MeanPool}(H_s), \quad z_s = W_{2,s}\,\text{GELU}(W_{1,s} h_s)\)$ Staging uses class-balanced cross-entropy (3-class softmax); phenotyping uses weighted BCE (12-label sigmoid).

3. Coordinator Agent: Optimizes weighted aggregation \(p = \sum_s \alpha_s p_s\) on the validation set using Bayesian optimization, Nelder-Mead, Differential Evolution, and gradient-based search to find \(\{\alpha_s\}\).

Key Contributions

• Clinically aligned expert decomposition matching real-world dementia assessment workflow (risk, cognitive, collateral, linguistic, visuographic domains).
• Domain-specific evidence routing with explicit missing-data preservation in textual representations.
• Probabilistic coordinator aggregation with multi-strategy weight optimization, yielding task-dependent (not fixed) expert weighting.
• First multi-agent framework evaluated on syndrome-level dementia staging and 12-phenotype classification on a real-world clinical cohort (1,066 patients, 2012–2024).

Results

• vs. best supervised monolithic MLLM: +8.7% staging macro-F1; +28.6% relative gain in phenotyping micro-F1.
• Full model (staging): AUC-Macro 0.874, Acc 0.757, F1-Macro 0.745.
• Full model (phenotyping): AUC-Macro 0.743, F1-Macro 0.242, F1-Micro 0.364, Hit@3 0.754.
• vs. MedAgentPro (best prior multi-agent): staging AUC 0.874 vs 0.689; phenotyping micro-F1 0.364 vs 0.308; Hit@3 0.754 vs 0.551.
• Zero-shot MLLMs degenerate to predicting all cases as "Dementia" (staging F1 as low as 0.109); few-shot Qwen3-VL achieves staging AUC 0.722 but no phenotyping results reported.
• Staging ROC: MCI shows lowest separability (intermediate clinical presentation); phenotyping AUPR correlates with label prevalence (Spearman \(\rho = 0.79\)).
• Ablation: removing Cognitive expert most degrades staging; removing Collateral expert most degrades phenotyping; uniform coordinator weighting reduces performance vs. optimized weights.

Limitations

• Private cohort only (1,066 patients from two sites in Australia, 2012–2024); no public benchmark or cross-institution validation.
• Long-tailed phenotype distribution: rare subtypes receive little training signal and low AUPR.
• MCI remains the hardest staging boundary (lowest ROC separability), inherently ambiguous clinically.
• Expert encoders (Qwen3-VL-Embedding-8B) are large; deployment on resource-constrained clinical hardware is not addressed.
• Coordinator weight optimization is validation-set-dependent; generalization of learned \(\{\alpha_s\}\) to new sites is untested.

Relevance to Harnesses / Meta-Harnesses

Dementia-Agents is a domain-specific instantiation of the meta-harness pattern: a coordinator agent that dynamically weights and aggregates outputs from specialized sub-agents, each fine-tuned on a distinct evidence modality. The three-step pipeline (route → specialize → aggregate) directly mirrors general-purpose meta-harness designs, making this a concrete case study of harness decomposition in a high-stakes structured prediction setting. The multi-strategy weight optimization in the coordinator (Bayesian, evolutionary, gradient-based) is an explicit harness-level hyperparameter search over agent mixture weights, which is a reusable design primitive. Practitioners building general medical or multi-modal meta-harnesses can draw on the routing-with-missing-data and probabilistic aggregation mechanisms demonstrated here.