Skip to content

MultiView-Bench: A Diagnostic Benchmark for World-Centric Multi-View Integration in VLMs

🕒 Published (v1): 2026-07-09 22:22 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MultiView-Bench is a diagnostic benchmark testing whether VLMs can integrate multiple viewpoints into a coherent, world-centric (allocentric) 3D scene representation grounded in a fixed global coordinate frame. Systematic evaluation of seven frontier VLMs reveals near-random performance on full 3D tasks, with the primary failure being axis direction identification rather than object recognition or 2D spatial reasoning. ViewNavigator, a multi-agent framework with probabilistic belief aggregation and active view selection, substantially improves all tested models even under a compute-matched budget.

Problem

Existing VLM benchmarks assess single-view or egocentric spatial reasoning (perspective-taking, camera-relative navigation, viewpoint transformation). No prior benchmark requires models to aggregate concurrent multi-view observations into a single, view-invariant global coordinate representation—the allocentric 3D mental model prerequisite for real-world tasks like mechanical assembly, CAD manipulation, and robotic placement. Single-view benchmarks permit multiple plausible scene configurations and cannot diagnose multi-view integration failures.

Method

Benchmark construction. A procedural Blender-based pipeline generates scenes with: (i) geometric primitives or real-world assets from 3DCoMPaT++; (ii) a fixed, visible global coordinate system (X=red, Y=green, Z=blue axes as 3D meshes invariant across cameras); (iii) six uniformly distributed azimuth viewpoints with slight elevation; (iv) controlled degree-of-freedom (DoF=1, 2, 3) variants isolating 1D, 2D, and full 3D reasoning demands. Models must output relative object positions as \(({\pm X}/0,\; {\pm Y}/0,\; {\pm Z}/0)\) in the global frame. The main benchmark comprises 5 task variants Ă— 100 instances each; 20 additional controlled variants probe biases.

ViewNavigator. A closed-loop multi-agent system with three components: 1. Belief Module: maintains a Dirichlet distribution per axis, aggregating VLM votes from micro-jittered viewpoints and down-weighting inconsistent predictions. 2. Active View Selection: an LLM planner chooses the next viewpoint to maximally reduce current belief uncertainty. 3. Confidence-Gated Output: halts when posterior probability per axis exceeds a threshold, preventing premature answers.

Budget-matched evaluation constrains ViewNavigator to \(\leq 6\) total viewpoints, no micro-jitter, and uniform belief updates—identical compute to the fixed-six-view baseline.

Key Contributions

  • MultiView-Bench: first benchmark explicitly requiring view-invariant, world-centric 3D integration from concurrent multi-view RGB images grounded in a visible global coordinate frame.
  • Extensible procedural data pipeline supporting arbitrary 3D assets, viewpoint configurations, and DoF control; 25 total task variants auto-generated with guaranteed ground-truth labels.
  • Systematic evaluation of seven frontier VLMs (Claude 3.7 Sonnet, Claude 4 Sonnet, Gemini 2.5 Flash, Gemini 2.5 Pro, GPT-4o, GPT-5, GPT-o3) exposing consistent failure modes and inductive biases.
  • ViewNavigator: a training-free, plug-and-play multi-agent scaffold using Dirichlet belief aggregation and active view selection.

Results

  • Frontier VLMs near random chance on 3D DoF=3 (random baseline \(\approx 3.7\%\)): most models including Claude series and GPT-4o score below 20%; GPT-5 (best model) reaches only ~50%.
  • Single isometric view: all models drop to random chance, confirming multi-view integration is essential.
  • DoF scaling: accuracy improves substantially as DoF decreases; Claude 3.7 Sonnet approaches GPT-5 at DoF=1.
  • Real-world objects (3DCoMPaT++): consistently lower than synthetic primitives across all models.
  • Failure localization: models succeed at Steps 1–2 (object ID, 2D spatial relation) but fail at Step 3 (axis direction identification in 3D context).
  • 2D decomposition: reframing 3D as three canonical 2D planes (XZ, YZ, XY) dramatically improves accuracy; splitting across independent single-view agents yields further gains.
  • Coordinate rotation bias: performance degrades severely at non-standard axis orientations (23°, 45°, 68°), revealing reliance on memorized right-hand-coordinate priors.
  • ViewNavigator (budget-matched): GPT-4o 2% → 19%; Claude 4 Sonnet 5% → 25%; GPT-5 49% → 61% on 3D DoF=3.
  • ViewNavigator (full, unconstrained, up to 10 viewpoints + micro-jitter): GPT-5 improves by 26% on 3D DoF=3; overall 3–5Ă— gains for the unconstrained agent across weaker models.

Limitations

  • Benchmark restricted to synthetic Blender scenes with clean geometry and visible axes; does not test real-world photorealistic multi-view inputs.
  • All axes remain visible across all viewpoints; real deployment may involve occluded or absent coordinate references.
  • Ground-truth labels are purely relative (sign of displacement), not metric—no distance estimation tested.
  • ViewNavigator requires access to a rendering engine that can produce arbitrary viewpoints on demand, limiting applicability to settings with controllable cameras.
  • Evaluation covers only seven models at a single point in time; newer models are not evaluated.
  • Diminishing returns of ViewNavigator at higher base-model performance suggests architectural limits remain unaddressed.

Relevance to Harnesses / Meta-Harnesses

ViewNavigator is a concrete multi-agent harness architecture: it wraps a base VLM with an active view-selection policy and a probabilistic belief aggregator that synthesizes evidence across queried viewpoints, demonstrating that scaffold-level decisions (which view to request next, how to merge partial observations) can recover substantial capability that the base model lacks entirely. The benchmark's finding that failure localizes to axis-direction identification — not perception or 2D reasoning — is directly actionable for harness designers: it tells you where to intervene with structured intermediate representations or explicit coordinate-frame reasoning rather than relying on the model's end-to-end forward pass. The compute-matched comparison validates that the performance gain is from scaffolding quality, not raw inference budget, which is the key question anyone building a meta-harness needs answered before committing to an orchestration strategy.