Skip to content

An Exam for Active Observers

🕒 Published (v1): 2026-07-17 17:46 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ActiveVision is a benchmark of 17 tasks across 3 cognitive categories designed to test whether multimodal LLMs can perform active observation — iteratively returning to visual evidence rather than encoding the image once. Frontier MLLMs collapse catastrophically: the best model (GPT-5.5 xhigh) achieves only 10.6% accuracy while three humans average 96.1%, and the gap persists even when models are given agentic tool-use with code execution.

Problem

Current vision-language benchmarks (MMMU-Pro, CharXiv, etc.) are near saturation and dominated by single-image VQA and captioning tasks that can be answered from a one-shot static description of the image. They do not measure active observation — the iterative, hypothesis-driven perceptual loop that psychophysics and cognitive science identify as essential for tasks requiring exhaustive scanning, sequential traversal along connected structures, or fine-grained cross-region attribute comparison. As a result, it is unknown whether MLLM benchmark gains reflect genuine progress on this capability.

Method

The authors construct ActiveVision using a two-stage generation pipeline:

  1. Procedural scaffold generation: A deterministic Python script (seeded for reproducibility) produces geometric instances with exact ground-truth labels — arbitrary continuous-coordinate positions, arbitrary Fourier-synthesized shape boundaries, and random smooth splines for traversal paths. Answer distributions are flat per task to block modal-answer shortcuts.
  2. Photorealistic re-rendering: GPT-image-2 converts each scaffold to a photorealistic image (e.g., Voronoi regions → aerial fields, closed loops → ropes on driftwood) using task-specific prompts that preserve counts, topology, and positions. Only the rendered image is shown to the evaluated model.

This yields 85 items (5 per task × 17 tasks). Evaluation uses exact-match accuracy after normalizing case/whitespace/separators, with the model's final answer extracted from <answer> tags under pure chain-of-thought (no system prompt). Additional ablations test reasoning-effort scaling across API tiers and agentic tool use (Codex, two Claude Code agents that write and execute vision code).

The 17 tasks span three cognitive axes: - Distributed Scanning (5 tasks): Region Counting, Connected Component Counting, Bounded Face Counting, Tangled Loop Counting, Singleton Shape Counting - Sequential Traversal (5 tasks): Arrow Chain Following, Traversal Point Ordering, Color Zone Sequencing, Line Intersection Sequencing, Maze Path Tracing - Visual Attribute Transfer (7 tasks): Constellation/Silhouette/Stroke Match Counting, Contour/Field/Signal/Stroke Difference Spotting

Key Contributions

  • ActiveVision benchmark: 17 procedurally generated, photorealistically rendered tasks that isolate active observation as a measurable behavioral capability distinct from static perception.
  • Task design principle: Every instance encodes discriminative visual state whose information content exceeds what any single language description can losslessly preserve, structurally forcing iterative image access.
  • Systematic evaluation of frontier MLLMs (GPT-5.5, Claude Fable 5, Claude Opus 4.7/4.8, Gemini 3.1 Pro, Gemini 3.5 Flash) across their full reasoning-effort API ranges.
  • Tool-use ablation: Tests Codex and Claude Code agents (write-and-run vision code), showing the bottleneck shifts to verification of tool failures rather than disappearing.
  • Failure mode taxonomy: Partial coverage and faulty individuation (Distributed Scanning); gestalt interpolation (Sequential Traversal); prior substitution (Visual Attribute Transfer).

Results

  • Best model accuracy: GPT-5.5 xhigh — 10.6% (9/85); scores zero on 11 of 17 tasks.
  • Human baseline: 3 participants averaging 96.1% (range: 94.1%–97.6%), ~9× the best model.
  • Other frontier models: Claude Fable 5 max — 3.5% (3/85); Gemini 3.5 Flash high — 8.2% (7/85); Gemini 3.1 Pro high — 5.9% (5/85); Claude Opus 4.8 max — 2.4% (2/85); Claude Opus 4.7 max — 4.7% (4/85).
  • Reasoning effort scaling: Scaling GPT-5.5 from none to xhigh raises cost ~100× but accuracy only from 2.4% → 10.6%. Spending 31× more per item on Fable 5 yields no accuracy gain (differences within sampling noise).
  • Question-only control: GPT-5.5 without the image solves 2/85 (2.4%), matching the none-effort image+question run, confirming prompt priors alone do not explain model performance.
  • Agentic tool use: Codex and Claude Code agents reach 24.7%–50.6% accuracy, gains concentrating on tasks with reliable code-based solutions; Sequential Traversal remains difficult. Tool failures on realistic imagery go undetected, shifting the bottleneck to perceptual verification within the agent loop.
  • Counting bias: Predicted vs. ground-truth count slopes are 0.44–0.69 across models, revealing systematic undercounting that worsens as true counts grow.
  • Sequential walk survival: No walk task is completed with all steps correct; prefix survival curves collapse within the first few steps across three traversal tasks.

Limitations

  • Small evaluation split: 85 items total (5 per task), making per-task and inter-model differences susceptible to sampling noise.
  • Human baseline is three participants: An N=3 human sample is insufficient to characterize variance or individual differences in active observation ability.
  • Rendering fidelity not verified: The photorealistic re-rendering step (via GPT-image-2) is assumed to preserve topology and counts; systematic errors introduced by re-rendering are not audited.
  • No architecture interventions: The benchmark diagnoses failure but does not propose or test architectural modifications (e.g., explicit spatial memory, iterative cropping) that might address the gap.
  • Tool-use evaluation is limited to code agents: Non-code tool-use paradigms (e.g., zoom-in, region-crop APIs) are not tested.
  • Text truncation in the paper: The provided text ends before §4.3 (failure mode analysis) and §5 are fully reproduced; some diagnostic depth is unavailable.

Relevance to Vision-Language Models

ActiveVision directly targets a structural limitation of current VLM architectures: images are encoded once into a fixed token sequence, with no explicit mechanism to re-query the visual representation as reasoning evolves. The benchmark operationalizes the psychophysics prediction that passive single-pass encoders will systematically fail tasks requiring iterative gaze redirection, providing a clean empirical signal that is orthogonal to existing near-saturated VLM benchmarks. For researchers tracking VLMs, the key finding is that neither scaling reasoning compute nor equipping models with code-execution tools closes the gap — active perceptual grounding is a distinct capability. This motivates VLM architectures with dynamic visual attention, spatial memory, or explicit image-query loops as a next frontier beyond chain-of-thought scaling.