Skip to content

MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems

🕒 Published (v1): 2025-05-25 02:32 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MetaMind is a three-stage multi-agent framework that decomposes social understanding into Theory-of-Mind hypothesis generation, moral/cultural refinement, and validated response synthesis. It achieves 35.7% average improvement on real-world social simulation tasks and matches human-level ToM performance on key benchmarks. The architecture is model-agnostic and validated across 16+ LLMs.

Problem

LLMs treat social reasoning as a single-step prediction problem, failing to capture the structured, multi-stage cognitive process humans use to infer unobservable mental states (beliefs, intentions, emotions) in indirect or culturally nuanced communication. Prior approaches—role-play prompting, RLHF, fine-tuning—optimize for surface-level alignment without layered reflection or norm-aware revision.

Method

MetaMind is a three-stage metacognitive pipeline:

  1. ToM Agent (Stage 1): Given input \(X = (u_t, C_t, M_t)\) (utterance, context, social memory), generates a hypothesis set \(H_t = \{h_1, \ldots, h_k\}\) over five mental-state types \(T = \{\text{Belief, Desire, Intention, Emotion, Thought}\}\) via commonsense reasoning cross-referenced against a dynamic social memory store.

  2. Moral Agent (Stage 2): Refines each \(h_i\) under constraint rules \(D\) (cultural norms, ethical constraints, role-based expectations) and selects the best hypothesis via: $\(\tilde{h}^* = \arg\max_i \left[\lambda \cdot P(\tilde{h}_i | u_t, C_t, M_t) + (1-\lambda) \cdot \log \frac{P(\tilde{h}_i | u_t, C_t, M_t)}{P(\tilde{h}_i)}\right]\)$ balancing contextual plausibility against information gain.

  3. Response Agent (Stage 3): Generates a response conditioned on \(\tilde{h}^*\) and social memory \(M_t\), then self-validates using a utility score \(U(o_t) = \beta \cdot \text{Empathy}(o_t, u_t, M_t) + (1-\beta) \cdot \text{Coherence}(o_t, C_t, \tilde{h}^*)\), triggering regeneration if the score is too low.

Social memory is a dynamic database of user emotional patterns and preferences updated after each interaction.

Key Contributions

  • Three-stage metacognitive multi-agent architecture formalizing hypothesis generation, norm-constrained refinement, and self-validating response synthesis.
  • Model-agnostic framework demonstrated to improve both open-source (Mistral, Qwen, Baichuan) and proprietary (GPT-3.5, GPT-4) LLMs.
  • First demonstration of LLMs matching average human performance on ToM benchmarks within a prompting framework (no fine-tuning).
  • Ablation study confirming all three stages and social memory are individually necessary.

Results

  • ToMBench (ToM reasoning): MetaMind + GPT-4 reaches 81.0% avg. accuracy vs. 74.8% base GPT-4 (+6.2%); surpasses CoT (73.6%), SymbolicToM (75.4%), Generative Agents (75.1%).
  • Social cognition tasks (8 tasks): 80.5% avg. vs. 71.5% base GPT-4 (+9.0%); largest gains in AST (+11.3%), SIT (+11.4%), FRT (+10.6%).
  • STSS social simulation: 73.9% avg. vs. 39.4% base GPT-4 (+34.5%); stated as 35.7% improvement in abstract; large gains in Appointment (+63.8 pp) and Inviting Companions (+64.8 pp).
  • Consistent gains across all 16 tested LLMs (Figure 2).
  • Ablation: removing Stage 2 causes the largest average drop on social cognition (–3.8%); removing Stage 3 causes the largest drop on STSS (–16.1%).

Limitations

  • No fine-tuned variant evaluated; framework relies entirely on prompting, making latency proportional to three sequential LLM calls plus optional regeneration.
  • Constraint rules \(D\) for the Moral Agent are hand-specified; coverage of edge cases across cultures is not characterized.
  • Social memory update mechanism details are deferred to an appendix; long-term memory drift or conflict resolution is not analyzed.
  • Human performance comparison is mentioned but the human-level threshold and evaluation protocol are not fully described in the excerpted text.
  • Hyperparameter sensitivity (\(k\), \(\lambda\), \(\beta\)) is deferred to an appendix rather than analyzed in the main body.

Relevance to Harnesses / Meta-Harnesses

MetaMind is a textbook staged multi-agent harness: it wraps arbitrary base LLMs in a fixed three-stage orchestration loop (generate → refine → validate/regenerate), exactly the pattern seen in meta-harnesses that add reasoning structure on top of frozen models. The self-validation loop in Stage 3 (utility-score-gated regeneration) is a lightweight inner harness that parallels the retry/verify loops common in agentic meta-harnesses. The model-agnostic, prompt-only design means the harness can be dropped onto any LLM backend without retraining, which is the defining property of a plug-and-play meta-harness. For researchers building harnesses, MetaMind demonstrates that domain-specific pipeline topology (here: ToM → moral filtering → empathic validation) outperforms generic CoT by large margins, reinforcing that harness architecture matters more than model scale for specialized tasks.