Towards Comprehensive Scene Understanding: Integrating First and Third-Person Views for LVLMs¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Current LVLMs deployed in egocentric (first-person) settings suffer from limited field of view and lack of global context, causing failures on spatially demanding queries. This paper introduces E3VQA, a 4K-pair benchmark for ego-exo multi-view VQA, and M3CoT, a training-free prompting method that fuses scene graphs from three complementary perspectives to improve multi-view reasoning. M3CoT yields +4.84% and +5.94% accuracy over the CCoT baseline on GPT-4o and Gemini 2.0 Flash respectively.

Problem¶

Egocentric video inputs—critical for AR/VR and embodied agents—have an inherently limited field of view and lack global scene context, causing LVLMs to fail on queries requiring spatial awareness beyond the immediate frame (e.g., object locations outside the field of view, counting people in the full scene). No benchmark existed to rigorously evaluate joint ego-exo reasoning in LVLMs, and existing CoT methods were not designed for multi-view settings.

Method¶

E3VQA Benchmark Construction: 4,000 multiple-choice QA pairs derived from Ego-Exo4D synchronized ego-exo video clips. A three-step automated pipeline generates single-view QA pairs, expands them with four input-condition responses (ego-only, exo-only, both, text-only), then filters out questions answerable without multi-view integration (retaining ~3.6% of 110,400 initial candidates). Four human annotators verify and craft distractors from the multi-condition responses to maximize difficulty. Four categories: pose/action perception, object/attribute perception, numerical reasoning, spatial reasoning (500 ego + 500 exo questions each).

M3CoT Prompting: Three LVLM agents each generate a scene graph from a distinct perspective: - Ego&Exo: both images processed jointly in one step - Ego2Exo: ego graph generated first, then extended with exo information - Exo2Ego: reverse order

These three scene graphs are then iteratively refined via multi-agent cross-pollination—each agent incorporates the other two agents' graphs to update its own—converging via majority voting at each iteration. The unified scene graph is provided as context for final answer generation.

Key Contributions¶

E3VQA: first benchmark for synchronized ego-exo multi-view VQA with 4K expert-verified QA pairs across four task categories
M3CoT: training-free, multi-agent CoT prompting technique that constructs unified scene graphs from three complementary ego-exo perspectives
Comprehensive evaluation of 5 closed-source and 9 open-source LVLMs on E3VQA, revealing consistent egocentric underperformance and specific failure modes in multi-view reasoning
Analysis showing that redundant multi-view context can hurt performance when a single view suffices, and that performance on true Both-view questions remains below 40% even for top models

Results¶

GPT-4o achieves 60.90% overall on E3VQA (best among all tested models); random chance would be 25%
M3CoT vs. CCoT baseline: +4.84% on GPT-4o (63.74% → 68.58%), +5.94% on Gemini 2.0 Flash (60.18% → 66.12%)
M3CoT vs. DDCoT: +4.15% (GPT-4o), +5.03% (Gemini); vs. CoCoT: +5.71% (GPT-4o), +5.81% (Gemini)
Numerical reasoning gains are largest: +6.88% (GPT-4o) and +8.13% (Gemini) over CCoT
Best open-source model: InternVL3-14B at 53.02%; Qwen-VL-Chat-7B scores only 28.13%
All models underperform on egocentric vs. exocentric questions across categories
"Both" subset (requiring true dual-view integration) stays below 40% accuracy for all models, even with M3CoT (53.04% for Gemini with M3CoT vs. 37.49% default)
Providing both views to models that only need one view (Ego/Exo subsets) causes accuracy drops, indicating context confusion from redundant inputs

Limitations¶

M3CoT is training-free but incurs significant inference overhead: three independent scene graph generation passes plus iterative multi-agent refinement, roughly 6–9× the inference cost of a single forward pass
Evaluation restricted to closed-source (GPT-4o, Gemini) and open-source 7B–14B models; M3CoT results on open-source models are deferred to appendix
E3VQA uses only still image frames sampled from video, not full video sequences, potentially missing temporal reasoning demands
All ego-exo pairs sourced from Ego-Exo4D, limiting diversity of recording conditions and activity types beyond what that dataset covers
Even with M3CoT, "Both" subset accuracy remains low (≤53%), indicating the fundamental multi-view fusion problem is far from solved
Benchmark is multiple-choice only, which may not capture open-ended reasoning failures

Relevance to Vision-Language Models¶

This work directly probes a gap in LVLM evaluation: how well these models handle multi-viewpoint inputs, which is critical for embodied AI, AR/VR assistants, and robotics applications where egocentric cameras are the primary sensor. The finding that providing additional views can hurt performance when queries are single-view-sufficient is a non-obvious result with practical implications for input design in real deployments. M3CoT's scene-graph-based multi-agent approach complements existing CoT and compositional reasoning literature (CCoT, DDCoT) by explicitly addressing view alignment rather than just sequential decomposition. The E3VQA benchmark fills a structural gap in the VQA ecosystem and provides a reusable testbed for future multi-view LVLM research.