Skip to content

MultiView-Bench: A Diagnostic Benchmark for World-Centric Multi-View Integration in VLMs

🕒 Published (v1): 2026-07-09 22:22 UTC · Source: Arxiv · link

Why this paper was selected

Multi-view integration benchmark; targets untested allocentric 3D reasoning gap in VLMs; diagnostic benchmark value

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MultiView-Bench is a diagnostic benchmark testing whether VLMs can integrate multiple viewpoints into a coherent, world-centric (allocentric) 3D scene representation grounded in a fixed global coordinate frame. Systematic evaluation of seven frontier VLMs reveals near-random performance on full 3D tasks, with the primary failure being axis direction identification rather than object recognition or 2D spatial reasoning. ViewNavigator, a multi-agent framework with probabilistic belief aggregation and active view selection, substantially improves all tested models even under a compute-matched budget.

Problem

Existing VLM benchmarks assess single-view or egocentric spatial reasoning (perspective-taking, camera-relative navigation, viewpoint transformation). No prior benchmark requires models to aggregate concurrent multi-view observations into a single, view-invariant global coordinate representation—the allocentric 3D mental model prerequisite for real-world tasks like mechanical assembly, CAD manipulation, and robotic placement. Single-view benchmarks permit multiple plausible scene configurations and cannot diagnose multi-view integration failures.

Method

Benchmark construction. A procedural Blender-based pipeline generates scenes with: (i) geometric primitives or real-world assets from 3DCoMPaT++; (ii) a fixed, visible global coordinate system (X=red, Y=green, Z=blue axes as 3D meshes invariant across cameras); (iii) six uniformly distributed azimuth viewpoints with slight elevation; (iv) controlled degree-of-freedom (DoF=1, 2, 3) variants isolating 1D, 2D, and full 3D reasoning demands. Models must output relative object positions as \(({\pm X}/0,\; {\pm Y}/0,\; {\pm Z}/0)\) in the global frame. The main benchmark comprises 5 task variants Ă— 100 instances each; 20 additional controlled variants probe biases.

ViewNavigator. A closed-loop multi-agent system with three components: 1. Belief Module: maintains a Dirichlet distribution per axis, aggregating VLM votes from micro-jittered viewpoints and down-weighting inconsistent predictions. 2. Active View Selection: an LLM planner chooses the next viewpoint to maximally reduce current belief uncertainty. 3. Confidence-Gated Output: halts when posterior probability per axis exceeds a threshold, preventing premature answers.

Budget-matched evaluation constrains ViewNavigator to \(\leq 6\) total viewpoints, no micro-jitter, and uniform belief updates—identical compute to the fixed-six-view baseline.

Key Contributions

  • MultiView-Bench: first benchmark explicitly requiring view-invariant, world-centric 3D integration from concurrent multi-view RGB images grounded in a visible global coordinate frame.
  • Extensible procedural data pipeline supporting arbitrary 3D assets, viewpoint configurations, and DoF control; 25 total task variants auto-generated with guaranteed ground-truth labels.
  • Systematic evaluation of seven frontier VLMs (Claude 3.7 Sonnet, Claude 4 Sonnet, Gemini 2.5 Flash, Gemini 2.5 Pro, GPT-4o, GPT-5, GPT-o3) exposing consistent failure modes and inductive biases.
  • ViewNavigator: a training-free, plug-and-play multi-agent scaffold using Dirichlet belief aggregation and active view selection.

Results

  • Frontier VLMs near random chance on 3D DoF=3 (random baseline \(\approx 3.7\%\)): most models including Claude series and GPT-4o score below 20%; GPT-5 (best model) reaches only ~50%.
  • Single isometric view: all models drop to random chance, confirming multi-view integration is essential.
  • DoF scaling: accuracy improves substantially as DoF decreases; Claude 3.7 Sonnet approaches GPT-5 at DoF=1.
  • Real-world objects (3DCoMPaT++): consistently lower than synthetic primitives across all models.
  • Failure localization: models succeed at Steps 1–2 (object ID, 2D spatial relation) but fail at Step 3 (axis direction identification in 3D context).
  • 2D decomposition: reframing 3D as three canonical 2D planes (XZ, YZ, XY) dramatically improves accuracy; splitting across independent single-view agents yields further gains.
  • Coordinate rotation bias: performance degrades severely at non-standard axis orientations (23°, 45°, 68°), revealing reliance on memorized right-hand-coordinate priors.
  • ViewNavigator (budget-matched): GPT-4o 2% → 19%; Claude 4 Sonnet 5% → 25%; GPT-5 49% → 61% on 3D DoF=3.
  • ViewNavigator (full, unconstrained, up to 10 viewpoints + micro-jitter): GPT-5 improves by 26% on 3D DoF=3; overall 3–5Ă— gains for the unconstrained agent across weaker models.

Limitations

  • Benchmark restricted to synthetic Blender scenes with clean geometry and visible axes; does not test real-world photorealistic multi-view inputs.
  • All axes remain visible across all viewpoints; real deployment may involve occluded or absent coordinate references.
  • Ground-truth labels are purely relative (sign of displacement), not metric—no distance estimation tested.
  • ViewNavigator requires access to a rendering engine that can produce arbitrary viewpoints on demand, limiting applicability to settings with controllable cameras.
  • Evaluation covers only seven models at a single point in time; newer models are not evaluated.
  • Diminishing returns of ViewNavigator at higher base-model performance suggests architectural limits remain unaddressed.

Relevance to Vision-Language Models

MultiView-Bench directly probes a fundamental VLM capability gap: the ability to form allocentric, view-invariant 3D representations from multi-view RGB inputs without geometric priors or depth maps. The finding that all frontier VLMs fail at axis direction identification—despite succeeding at 2D spatial reasoning—pinpoints a specific architectural or training deficiency rather than a general intelligence limitation, providing a precise diagnostic target for future VLM development. The benchmark's controlled DoF design enables fine-grained ablation that isolates where in the 3D reasoning chain current models break, which is directly actionable for dataset curation, training objective design, and model architecture decisions. ViewNavigator demonstrates that training-free agentic scaffolding with probabilistic belief aggregation can recover substantial performance, offering a near-term practical scaffold while pointing to the depth of representation learning improvements still needed.