Skip to content

SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding

🕒 Published (v1): 2026-07-11 17:02 UTC · Source: Arxiv · Venue: COLM 2026 · link

Why this paper was selected

COLM 2026; controlled benchmark for long-context VLM document understanding, fills real eval gap

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SynthDocBench is a fully synthetic, controlled benchmark for long-context visual document understanding that independently varies document length, layout archetype, modality composition, and question type across 200 reports (avg. 51.1 pages) and 1,788 questions. Evaluating eight frontier VLMs, it surfaces three systematic failure modes—sharp degradation with evidence complexity, positional sensitivity peaking in the document's middle third, and chart comprehension collapse in long-document contexts—that existing benchmarks cannot isolate. Results suggest frontier models are overfitting to benchmark artifacts rather than achieving robust long-context understanding.

Problem

Existing document VQA benchmarks (DocVQA, ChartQA, MMLongBench-Doc) either evaluate charts in isolation from document context or use real documents where length, layout, modality, and question difficulty co-vary uncontrollably. When a VLM fails on a real multi-page document, it is impossible to attribute that failure to a specific cause. Additionally, single-page benchmarks are approaching saturation (>95% on DocVQA, >89% on ChartQA for frontier models), providing no diagnostic separation. Long-context benchmarks like MMLongBench-Doc (best model: 57%) prioritize coverage breadth over controlled diagnosis.

Method

Pipeline (3 stages): 1. Document generation: A topic seed \(\tau\) is mapped through LLM-based content extraction to a structured intermediate representation. A layout archetype \(a \in A\) is sampled with \(p=0.6\) from a topic-conditioned distribution and \(p=0.4\) uniformly at random (the 40% override prevents layout-topic correlations). Six archetypes (Academic, Editorial, Dashboard, Magazine, Infographic, Brutalist) govern page grammar and chart placement. Each visualization is generated in a dual-layer form: a visible D3.js rendering and a hidden structured metadata object \(V_k\) encoding axes, data points, and derived statistics—enabling deterministic ground-truth derivation without pixel parsing. Reports are assembled as HTML and rendered to PDF via Playwright.

  1. QA generation: The pipeline parses documents into text, table, and visualization channels, then generates three question families—chart-reading (24 chart types, exact numerical reads), cross-modal (joint reasoning over charts and textually distant corroborating evidence), and complex multi-hop (2–4 evidence units, difficulty L1–L5 from direct value lookup to cross-section synthesis). A validation stage filters malformed or ambiguous items.

  2. Vision-only evaluation: Models receive only rasterized page images (144 DPI, 5-page vertical strips, max 7,900 px, 4 MB). Predictions scored by GPT-5 as judge (0–10), with threshold accuracy \(\text{ACC}(f_\theta) = \frac{1}{|Q_\text{valid}|} \sum_{q} \mathbf{1}[J(\hat{a}_q, a^*_q) \geq 6]\). Cross-judge validation against Gemini-3.1-Pro and Claude-Sonnet-4.5 as alternative judges confirms agreement within 3.5 ACC points (\(r \geq 0.94\)).

Key Contributions

  • SynthDocBench: 200 synthetic reports, 1,788 questions across three subsets (chart-reading, cross-modal, complex multi-hop), with independently variable length, depth, modality, and question-type axes; covers 24 chart types and 6 layout archetypes.
  • Dual-layer generation pipeline: Every chart produced simultaneously as a D3.js rendering and a structured metadata object, making ground-truth answers deterministic by construction with no post-hoc annotation or chart parsing.
  • Controlled combinatorial design: 40% random layout override and topic-conditioned archetype sampling prevent spurious correlations between document subject matter and visual structure.
  • Systematic failure mode characterization: Three concurrent, previously unobservable failure modes in frontier VLMs revealed through stratified analysis.

Results

  • Overall ranking: Gemini-3.1-Pro (ACC 0.725) > Qwen3.5-VL-122B (0.655) > Qwen3-VL-235B (0.586) > GPT-5.4 (0.423) > GPT-4o (0.386) ≈ InternVL3-78B (0.383) > Claude-Sonnet-4.5 (0.314) >> Qwen2.5-VL-7B (0.081).
  • Failure mode 1 — Complexity degradation: All models except Gemini degrade monotonically L1→L5; Claude-Sonnet-4.5 drops 23 pp, Gemini remains near-flat (0.670–0.784). Widest cross-model gap in technical/quantitative complex questions (Gemini: 0.643 vs. GPT-4o: 0.111).
  • Failure mode 2 — Positional sensitivity: The middle third of a document is hardest for 5 of 8 models (drop of 5–18 pp below Early); Qwen3.5-VL-122B shows steepest Early→Middle drop (−18.5 pp); 5 of 6 models show negative Early→Late trend (steepest: −8.3 pp per abstract; −11.7 pp for Claude-Sonnet-4.5 per main text).
  • Failure mode 3 — Chart comprehension collapse: OCR+GPT-4o achieves ACC 0.297 on chart-reading vs. vision GPT-4o 0.457, confirming chart questions require pixel-level visual decoding; the 46 pp gap between OCR chart ACC and Gemini's (0.759) isolates visual perception as a primary bottleneck.
  • OCR vs. vision asymmetry: Complex multi-hop questions are largely text-recoverable (OCR ACC 0.798 vs. vision 0.360); chart-reading reverses this (OCR 0.297 vs. vision 0.457).
  • Cross-modal subset is hardest for all models; 109 questions where all 8 models score ≤3 are dominated by cross-modal failures.
  • Benchmark correlation: Model rankings on SynthDocBench correlate with MMLongBench-Doc (Spearman \(\rho=0.657\), Pearson \(r=0.683\)), supporting external validity.

Limitations

  • Fully synthetic documents may not reflect all distributional properties of real-world corpora (heavy-tailed length, heterogeneous formatting, scanned artifacts).
  • A potential distribution-familiarity confound for Gemini-3.1-Pro is acknowledged but unresolved—the 7.0 ACC point lead may partially reflect training data overlap with the generation pipeline.
  • Page rasterization capped at 120 pages and 5-page vertical strips; denser strips improve performance where API limits permit, but cross-model comparability required this constraint.
  • Only 200 documents; tight unimodal distributions (by design) limit generalization claims to heavy-tailed real-world settings.
  • Visual hallucination error analysis is qualitative; no causal attribution to specific architectural components.

Relevance to Vision-Language Models

SynthDocBench directly addresses a core diagnostic gap for VLMs: the inability to attribute long-context document failures to specific factors when real documents confound length, modality, layout, and question complexity. The three uncovered failure modes—positional bias (lost-in-the-middle effect persisting in multimodal settings), complexity degradation across L1→L5, and chart comprehension collapse in document context—provide concrete, targeted targets for future VLM architectural and training improvements. The benchmark's finding that chart understanding isolated on ChartQA (>89%) collapses in long-document contexts is particularly significant for the VLM field, as it challenges the validity of isolated chart benchmarks as proxies for real-world capability. The controlled synthetic design also establishes a methodological template (analogous to CLEVR and bAbI in their domains) for the emerging field of long-context multimodal evaluation.