Skip to content

MeetingToM: Evaluating Multimodal LLMs on Theory-of-Mind Reasoning in Multi-Party Meetings

🕒 Published (v1): 2026-07-21 16:05 UTC · Source: Arxiv · link

Why this paper was selected

Multi-party ToM benchmark; measures social/epistemic reasoning gap in MLLMs for agentic settings

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MeetingToM is a multimodal benchmark that evaluates MLLMs on Theory-of-Mind (ToM) reasoning in naturalistic four-person product-design meetings, targeting latent social phenomena such as pseudo-consensus—verbal agreement that masks private dissent under social pressure. It organizes tasks hierarchically across subject, dyadic, and group levels of social granularity. Systematic evaluation reveals persistent failures in nonverbal integration, hidden-attitude inference, and pseudo-consensus detection across both proprietary and open-source models.

Problem

Existing multimodal ToM benchmarks focus on overt, externally verifiable signals in dyadic or narrative-centric settings, providing limited coverage of latent group dynamics, meeting-specific conformity effects, and the incongruence between verbal affirmations and nonverbal dissent. No prior benchmark explicitly targets pseudo-consensus or requires joint reasoning over addressee identification, attitude inference, and group consensus quality in a unified multi-party, multi-view video setting.

Method

MeetingToM is built on the AMI Meeting Corpus (synchronized four-participant product-design meetings with word-level transcripts and multi-view video). Three hierarchically structured multiple-choice VQA tasks are defined:

  • Task 1 – Subject-level mental state prediction: Given a 5-second close-up clip \(V^A\) and utterances \(U\), predict \(\hat{s}_A = \arg\max_s p(s \mid V^A, U, Q_\text{ms})\) over seven cognitive states (Active Engagement, Supportive Endorsement, Focused Listening, Cognitive Conflict, Confused Bewilderment, Hesitation, Disengaged Withdrawal).
  • Task 2 – Dyadic-level referential reasoning: (2.1) Addressee identification from global view \(V^g\) and \(U\) over \(R = \{A, B, C, D, \text{Multiple}, \text{Unknown}\}\); (2.2) Attitude inference from mosaic view \(V^m\) over \(M = \{\text{Support}, \text{Oppose}, \text{Neutral}, \text{Uncertain}\}\).
  • Task 3 – Group-level consensus reasoning: (3.1) Consensus classification from \(V^m\) and \(U\) over \(C = \{\text{TC}, \text{PC}, \text{NC}, \text{Uncertain}\}\); (3.2) Conditional dissenter identification if \(\hat{c} = \text{PC}\).

The mosaic view \(V^m\) concatenates four synchronized close-up feeds into a 2×2 panel. Annotation uses a hybrid pipeline: GPT-5 generates candidate questions/options (without determining gold answers), two trained human annotators independently label each instance, and a third adjudicates disagreements with majority-vote filtering. Annotation quality is measured by Fleiss' \(\kappa\) on 200-instance subsets per task.

Key Contributions

  • Introduces MeetingToM: 1,800 video clips from 60 AMI meeting sessions (600 instances per task family) with synchronized multi-view video and transcripts.
  • Formalizes pseudo-consensus as a meeting-specific ToM target, distinguishing it from true consensus and absent consensus.
  • Provides a three-level hierarchical task design (subject → dyadic → group) with increasing relational reasoning demands.
  • Delivers a unified evaluation protocol with modality ablation, prompting strategy comparison, and Macro-F1 for class-imbalanced tasks.
  • Benchmarks GPT-4o, GPT-5, Gemini-3 Pro, Qwen2.5-VL-32B, Qwen3-VL-8B, and Qwen3-VL-32B with human performance as reference ceiling.

Results

  • Human vs. model gap: Human accuracy ranges 79.67–86.33 across tasks; best model Task-1 accuracy is 59.00 (Gemini-3 Pro) vs. 86.33 human.
  • Task 1: Gemini-3 Pro leads (59.00 Acc / 30.39 Macro); GPT-5 achieves 55.67 Acc / 28.69 Macro; Macro scores reveal severe class-imbalance inflation of raw accuracy.
  • Task 2.1 (Addressee ID): Gemini-3 Pro 74.33 Acc; Qwen3-VL-32B competitive at 64.67; combined video+utterance outperforms utterance-only here.
  • Task 2.2 (Attitude inference): Gemini-3 Pro 51.67; GPT-5 51.00; GPT-4o 46.15.
  • Task 3.1 (Consensus classification): GPT-5 best at 43.00 Acc / 28.99 Macro; Gemini-3 Pro 40.67 / 22.13.
  • Task 3.2 (Dissenter ID, conditional): GPT-5 90.70; Gemini-3 Pro 88.52; strong performance reflects conditioning on correctly identified pseudo-consensus cases.
  • Modality ablation (Gemini-3 Pro): Utterance-only outperforms video+utterance on Tasks 1, 2.2, 3.1, 3.2; combined beats utterance-only only on Task 2.1, indicating inadequate multimodal fusion.
  • Prompting: Evidence-grounded CoT (ECoT) yields +2.33 Task-1 accuracy and +14.33 Task-3.1 accuracy for Gemini-3 Pro; Task-CoT on Qwen3-VL-32B improves Task 1 by +15.33 but degrades Task 2.1 by −4.67.

Limitations

  • Benchmark sourced exclusively from AMI (English-language, four-person, product-design scenario), limiting generalizability to other meeting types, languages, and group sizes.
  • Task 3.1 is dominated by True Consensus labels, creating class imbalance that inflates accuracy metrics; minority-class performance is substantially lower.
  • GPT-5 is evaluated using static frames sampled at 0.5 FPS rather than native video, which may underestimate its temporal reasoning capability or introduce an artifact.
  • Task 3.2 evaluation is conditioned on Task 3.1 correctness, so dissenter-identification metrics do not reflect end-to-end pipeline performance.
  • Inter-annotator agreement for attitude inference (κ = 0.5014) and consensus classification (κ = 0.5681) is moderate, reflecting genuine task ambiguity but also posing a ceiling question.

Relevance to Vision-Language Models

MeetingToM directly stress-tests a core VLM capability—grounded multimodal fusion—by requiring models to reconcile nonverbal visual signals (gaze, posture, facial expression) with verbal transcripts in temporally extended, multi-speaker video. The finding that utterance-only inputs frequently match or exceed joint video+utterance performance exposes a concrete failure mode in current VLMs: they cannot reliably extract and leverage fine-grained visual behavioral cues even when they are decision-critical. The pseudo-consensus task is particularly diagnostic because it demands detecting incongruence between modalities rather than reinforcing agreement, a regime where visual grounding must override or qualify verbal surface signals—precisely the kind of multimodal integration VLMs are theoretically designed for but empirically fail at.