Skip to content

MindPower: Enabling Theory-of-Mind Reasoning in VLM-based Embodied Agents

🕒 Published (v1): 2025-11-28 10:24 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MindPower introduces a Robot-Centric benchmark and training framework for Theory-of-Mind (ToM) reasoning in VLM-based embodied agents. It defines a 6-layer reasoning hierarchy (Perception→Belief→Desire→Intention→Decision→Action) and a GRPO-based Mind-Reward that enforces BDI consistency, enabling agents to simultaneously model their own and humans' mental states before generating actions. A fine-tuned Qwen2.5-VL-7B surpasses GPT-4o by 12.77% on decision accuracy and 12.49% on action generation.

Problem

Existing ToM benchmarks are role-centric — they ask models to infer a human character's mental state via multiple-choice questions, but never require the agent to reason from its own perspective or link that reasoning to executable actions. VLMs therefore remain reactive: they describe what they see but cannot detect false beliefs, infer implicit goals, or generate coherent assistance behaviors. This breaks down in embodied settings where an agent must proactively help a human whose beliefs may diverge from reality.

Method

MindPower Benchmark (590 scenarios, 2 simulators — VirtualHome and ThreeDWorld, 8 home layouts, 16 humanoid agents): two tasks assess Robot-Centric ToM — False-Belief Correction (agent detects and resolves a human's mistaken object-location belief) and Implicit Goal Inference & Completion (agent infers an unstated goal from behavioral cues such as searching). Each example is annotated at all six hierarchy layers; Stage 3 of each video shows the human returning/searching, giving the agent a grounded trigger for ToM inference.

MindPower Reasoning Hierarchy: structured chain <Perception>→<Belief>→<Desire>→<Intention>→<Decision>→<Action>, where Belief models second-order reasoning (robot's belief about the human's belief), not just first-order.

Mind-Reward (GRPO training on Qwen2.5-VL-7B-Instruct): each reasoning layer is mapped to an atomic action sequence action(agent, object). Reward combines: - Atomic Accuracy (ROUGE-1, perspective-tagged): R_atomic - Local Consistency (ROUGE-2 on adjacent pairs): R_local - Global Consistency (ROUGE-L): R_global - Format-Reward (binary sequential regex match over all 6 tags)

Final reward: R = (0.2·R_atomic + 0.3·R_local + 0.5·R_global) + R_format. Training uses SFT cold-start (5 epochs) followed by GRPO (400 iterations, 8 generations/sample, single H800 GPU).

Key Contributions

  • Robot-Centric ToM benchmark with 590 open-ended scenarios requiring both decision-making and executable action generation — the first to link mental reasoning to embodied actions.
  • MindPower Reasoning Hierarchy: a standardized 3-level, 6-layer BDI chain that bridges perception to action and supports second-order belief reasoning.
  • Mind-Reward: a ROUGE-based GRPO reward over atomic action sequences enforcing ToM consistency across all reasoning layers.
  • Empirical demonstration that structured ToM reasoning hierarchy substantially outperforms standard chain-of-thought (<think>) on decision and action tasks.

Results

  • Ours vs. base Qwen2.5-VL-7B-Instruct: +20.04% Sentence Transformer score on Perception; +23.33% on Decision layer; SR +11.6 pp; AC +15.25 pp.
  • Ours vs. GPT-4o: +12.77% decision accuracy; +12.49% action generation accuracy (paper abstract and contributions section).
  • Ablation — removing hierarchy from GPT-4o: decision accuracy −1.24%; action generation 2.91%→0.82%.
  • Standard <think> vs. MindPower hierarchy on GPT-4o: decision −4.89%; action 2.91%→0.90%.
  • SFT-only vs. SFT+Mind-Reward: SR 8.50→11.75; AC 10.48→15.40 — Mind-Reward adds consistent gains over SFT alone.
  • Gemini-2.5 Pro and GPT-4o lead closed-source baselines; Video-R1 and VideoChat-R1 lead open-source.
  • Human participants outperform all VLMs on every metric.

Limitations

  • Dataset is small (590 examples total; ~118 test) and confined to household simulation; real-robot generalization untested.
  • Only Qwen2.5-VL-7B-Instruct used as the trainable backbone; generalization of Mind-Reward to other VLMs unverified.
  • Action space is high-level and simulator-specific; no bridging to low-level motor control.
  • Large gap between human performance and all VLMs persists even after fine-tuning (human SR ~19.37 vs. ours ~11.75 on the full set).
  • Multi-agent coordination scenarios not included; benchmark does not cover deceptive or conflicting-goal interactions.

Relevance to Agentic AI / LLM Agents

MindPower tackles a foundational bottleneck for autonomous agents: acting correctly requires modeling why other agents behave as they do, not just what they do. The Robot-Centric BDI hierarchy is a direct instantiation of classical agent architectures (Belief-Desire-Intention) inside a neural VLM, and the Mind-Reward mechanism shows how RL can enforce multi-step reasoning coherence rather than just final-answer correctness. For practitioners building LLM agents that must collaborate with or assist humans — assistive robots, social agents, household AI — this paper provides both a concrete benchmark and a training recipe. The finding that standard chain-of-thought underperforms structured ToM prompting has immediate implications for prompt engineering in agentic pipelines.