Read the Room: Video Social Reasoning with Mental-Physical Causal Chains¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Video social reasoning benchmark with mental-physical causal chains; VLM ToM

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

R³-Bench is a video social reasoning benchmark with fine-grained annotations of belief, intent, desire, emotion, and their mental-physical causal chains, targeting Theory of Mind capabilities in LVLMs. R³-FDT is a companion large-scale training set (2.8k videos, 41k QA pairs) generated via an automated pipeline over movie data. Evaluation reveals SOTA LVLMs fall substantially short of human-level multi-step social reasoning; fine-tuning a 7B model on R³-FDT yields notable improvements across related benchmarks.

Problem¶

Existing video QA benchmarks either focus on factual/visual comprehension (MVBench, Video-MME) or cover only a narrow subset of mental states without multi-step causal structure (MMToM-QA, IntentQA, Social-IQ). No dataset provides: (i) all four key mental state types (belief, intent, desire, emotion) simultaneously, (ii) explicit mental-physical causal chains linking states across time, and (iii) sufficient scale (prior best training sets reach only ~6k QA pairs) to train foundation models for social reasoning.

Method¶

R³-Bench is constructed via a five-stage human-expert pipeline: (1) crowdworker video collection with strict causal/mental-state requirements; (2) expert verification; (3) expert annotation of mental-physical causal chains represented as directed graphs with event and mental-state nodes; (4) GPT-4o QA generation from chain structure following four question types; (5) expert QA verification. It yields two splits: R³-Bench-Hard (human-designed QA) and R³-Bench-DX (chain-derived QA). Four question types are defined: Event Understanding (EU), Mental State Estimation (MSE), Causal-Why (CW, abductive), and Causal-How/What (CH/W, deductive). A causal chain \(g \in G\) consists of subchains \(g_\text{sub}\); each subchain has one result node \(n_1\) and reason nodes \(\{n_{0i}\}\) connected by causal edges \(\{e_i\}\).

R³-FDT uses an automated pipeline over movie data (MovieNet, MovieQA, CondensedMovies): aligned clips with scripts and annotations are passed to GPT-4o to infer causal chains, followed by self-correction (removing ~6% of chains via consistency checks and redundancy removal) and QA/distractor generation. Distractors are constructed to be either common-sense plausible but ungrounded in the video, or video-grounded but causally incorrect. The pipeline avoids direct video-to-LLM reasoning to mitigate cross-modal hallucination.

Key Contributions¶

R³-Bench: 5.1k expert-verified QA pairs across 0.3k real-world videos with full mental-physical causal chain annotations covering belief, intent, desire, and emotion.
R³-FDT: 41k QA pairs over 2.8k movie-sourced training videos generated via a scalable automated pipeline with self-correction and hallucination detection.
Systematic LVLM evaluation demonstrating a large gap between SOTA model performance and human-level social reasoning on R³-Bench.
A fine-tuned 7B model trained on R³-FDT that achieves notable improvements across multiple social reasoning benchmarks.
First dataset with complete coverage of all four mental state types plus explicit causal chains at training scale.

Results¶

The paper text provided is truncated before the quantitative results section; specific accuracy numbers and per-model comparisons against baselines (e.g., GPT-4o, Gemini 1.5 Pro, LLaVA-family models) are not available in the supplied excerpt. The abstract and introduction state: - SOTA LVLMs show "substantial deficiencies in consistent multi-step social reasoning" relative to humans on R³-Bench. - Fine-tuning a 7B model on R³-FDT achieves "notable improvements across multiple relevant benchmarks." - Data pollution prevention: samples answerable by Gemini 1.5 Pro were excluded from R³-Bench-Hard, ensuring non-trivial difficulty.

Limitations¶

R³-FDT training data is sourced entirely from movies, which may not fully represent the domain distribution of real-world social interactions tested in R³-Bench.
R³-Bench test set is relatively small (5.1k QA, 0.3k videos), limiting statistical power for fine-grained subcategory analysis.
QA generation for R³-FDT relies on GPT-4o over text descriptions rather than direct video understanding, potentially missing purely visual cues not captured in scripts/annotations.
Self-correction removes only 6% of automated chains; residual noise in R³-FDT is likely non-negligible.
Evaluation scope is restricted to multiple-choice QA, which does not fully measure open-ended generative social reasoning.

Relevance to Vision-Language Models¶

This work directly targets a known weak spot of LVLMs: Theory-of-Mind-grounded, multi-step causal reasoning over video, where purely perceptual or factual benchmarks fail to stress-test the models. The structured causal chain annotation (belief → intent → emotion → action) provides a richer diagnostic signal than single-question benchmarks, enabling analysis of reasoning consistency across linked states—a quality distinct from accuracy on isolated questions. R³-FDT's scalable automated generation strategy is a practical template for producing hard social-reasoning training data without prohibitive annotation cost, directly relevant to post-training pipelines for video LVLMs. The gap between SOTA LVLM performance and human-level on R³-Bench establishes a clear capability frontier for the community to close.