MAMA-Memeia! Multi-Aspect Multi-Agent Collaboration for Depressive Symptoms Identification in Memes¶

🕒 Published (v1): 2025-12-31 18:06 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

MAMA-Memeia is a multi-agent, multi-aspect inference framework that uses three heterogeneous LLMs (GPT-4o, Claude 3.5 Sonnet, Gemini-2.0-flash), each specializing in a distinct knowledge aspect derived from Cognitive Analytic Therapy (CAT), to classify fine-grained depressive symptoms in social-media memes. The authors also release RESTOREx, a cleaned and human-explanation-augmented version of the RESTORE meme dataset. The framework achieves 72.73% macro-F1, a +7.55% gain over the previous state-of-the-art.

Problem¶

Multimodal memes are a prevalent channel for expressing depression, but their analysis is hampered by sarcasm, metaphor, and cross-modal irony that confound both unimodal and standard multimodal models. Existing work on the RESTORE dataset relies on fine-tuned ResNet+BERT fusion and does not exploit the reasoning depth of modern LLMs or clinical psychology frameworks. The prior dataset also contained annotation errors, non-meme images, and an unreliable label class, all of which degraded benchmark reliability.

Method¶

RESTOREx dataset construction: The RESTORE dataset is re-annotated (Krippendorff's α = 0.833), non-meme images (~20% reduction) and the unreliable "Lack of Energy" class are removed, and human-written gold explanations are added for all 520 test samples. LLM-generated explanations from six MLLMs (LLaVA 1.5, LLaVA-NeXT, MiniCPM-V, GPT-4o, Claude 3.5 Sonnet, Gemini-2.0-flash) are provided for training.

MAMA-Memeia framework (three phases): 1. Independent Ideation — each of the three LLM agents receives a distinct aspect-specific system prompt derived from CAT competencies: Depression Knowledge (Pd), Emotional Knowledge (Pe), or Cultural Knowledge (Pc). Each agent independently produces symptom predictions, scalar confidence scores per symptom ∈ [0,1], and a free-text explanation. 2. Collaborative Discussion — over R rounds, each agent receives the other agents' predictions, confidence scores, and explanations as a discussion prompt appended to its conversation history, then revises its outputs. 3. Consensus Resolution — final labels are determined by a threshold t on the mean confidence across agents: label j is predicted iff (1/n)Σ c_{i,j}·𝟙{j ∈ p_i} > t, averaging out over-prediction (Gemini) and under-prediction (GPT-4o) tendencies.

Key Contributions¶

RESTOREx dataset: re-annotated 7-label meme dataset (~7,926 samples) with human-written explanations for the test split and LLM-generated explanations for training.
CAT-grounded multi-aspect prompting: three aspect-specific prompts (Pd, Pe, Pc) derived systematically from eight Cognitive Analytic Therapy competency criteria.
MAMA-Memeia framework: heterogeneous-LLM multi-agent debate with confidence-weighted consensus, achieving new SOTA of 72.73% macro-F1 on RESTOREx.
Human evaluation of LLM explanations across five RQs (fluency, relevance, figurative understanding, persuasiveness, appeal), establishing closed-source superiority.

Results¶

MAMA-Memeia: 72.73% macro-F1, 72.45% weighted-F1 on RESTOREx.
Previous SOTA (Yadav et al. 2023, multimodal gating fusion): 65.18% macro-F1 → +7.55% improvement.
Best single-agent closed-source (Gemini-2.0-flash, combined Pd+Pc+Pe): 66.05% macro-F1 — MAMA-Memeia adds ~6.7 pp over this.
Human-written explanation baseline: 65.00% macro-F1 — MAMA-Memeia exceeds humans by ~8% weighted-F1.
Best unimodal text baseline (MentalBERT + explanations): 64.62% macro-F1.
Removing aspect-specific prompting ablation: 71.88% macro-F1 (−0.85 pp).
LLM explanations as training signal: BART OCR 44.71% → BART+LLM explanations 55.81% macro-F1 (+11.1 pp).

Limitations¶

Relies exclusively on closed-source LLMs (GPT-4o, Claude 3.5 Sonnet, Gemini-2.0-flash), limiting reproducibility and transparency in a sensitive mental health domain.
Multi-agent inference with R rounds of debate is computationally expensive per sample.
Gemini's systematic over-prediction and GPT-4o's under-prediction are observed but not mechanistically explained due to the black-box nature of these models.
The dataset covers only English-language memes from Twitter/Reddit; cross-lingual and cross-platform generalization is untested.
The system identifies symptoms in meme content, not in individuals — clinical deployment would require substantial expert oversight.
Dataset size remains modest (7,926 total samples); class imbalance is significant (e.g., LOI: 544 vs. ED: 1,853 train samples).

Relevance to Agentic AI / LLM Agents¶

This paper is a concrete instantiation of multi-agent LLM debate applied to a real-world classification task with domain-grounded role specialization — each agent holds a distinct knowledge persona (clinical, emotional, cultural) rather than being identical, which is a key design principle for diversity-driven multi-agent systems. The confidence-sharing and iterative revision protocol operationalizes ideas from frameworks like ReConcile (Du et al. 2024) in a multimodal, safety-critical setting. For researchers tracking agentic AI, this demonstrates both the gains and the remaining challenges (cost, opacity, over-/under-prediction tendencies) of heterogeneous LLM ensembles acting as collaborative agents, particularly when consensus mechanisms must counterbalance individual agent biases.