Skip to content

What Does Your Short-Answer VQA Score Actually Measure? Evaluator-Dependent Instability in Multimodal Short-Answer Benchmarks

🕒 Published (v1): 2026-07-11 10:01 UTC · Source: Arxiv · link

Why this paper was selected

Demonstrates VQA benchmark scores are evaluator-dependent; critical for anyone running VLM evals

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Short-answer VQA benchmarks measure a mixture of semantic correctness and surface-form compliance with the evaluator's answer contract, not semantic correctness alone. A human-validated LLM judge (97.6% precision) auditing 37k+ official errors across six VLMs and six benchmarks finds that up to 48% of flagged errors on text-rich benchmarks are semantically acceptable answers penalized purely for surface-form mismatch. The instability is structured by answer type and is further amplified by benign prompt rewrites that flip 6–23% of item-level outcomes without changing the underlying task.

Problem

Modern VLMs generate free-form natural language, but short-answer VQA benchmarks still evaluate via exact-match or ANLS string comparison against reference answers — a design suited for fixed-vocabulary prediction, not fluent generation. This means the official score \(S_\text{off}(o) = f(S_\text{sem}(o), \kappa(o))\) conflates two distinct quantities: semantic acceptability \(S_\text{sem}(o)\) and benchmark-contract compatibility \(\kappa(o)\). The resulting undercount contaminates official error pools, makes adjacent leaderboard gaps uninterpretable (mean judge correction on ST-VQA of +9.2 pp exceeds the tightest adjacent-model gap of 1.4 pp), and causes prompt-phrasing artifacts to register as task-level score differences.

Method

The paper introduces an operational answer-contract decomposition that assigns benchmark items into four heuristic buckets by question form, answer shape, and benchmark-native metadata: - Scalar: numeric, boolean, tightly closed answers - Extractive-readout: OCR-style span extraction and text readout - Identifier-like: codes, alphanumeric strings, uppercase identifiers - Compositional-multispan: list or conjunctive multi-span lookups

Benchmark undercount is quantified via two complementary layers: (1) a conservative heuristic (RA-Eval) that checks whether a gold reference \(g_\text{norm}\) appears as a word-boundary substring of the model output \(o_\text{norm}\) (with \(|g_\text{norm}| > 3\) characters), and (2) a text-only LLM-as-judge (DeepSeek-Chat) prompted for binary semantic equivalence on all 36,357 open-weight official errors. The judge is validated against human majority labels on 570 stratified items (precision 97.6%, recall 95.5%, Cohen's \(\kappa = 0.917\)) and replicated with a second judge (moonshot-v1-8k; per-dataset FN-rate \(r = 0.999\)). Perturbation sensitivity is measured with seven benign variants — five instruction-prefix rewrites (A1–A5) and two context-formatting edits (B1–B2) — applied to four open-weight VLMs across all six datasets.

Key Contributions

  • Formal decomposition of official VQA scores into semantic acceptability and evaluator contract compatibility, providing a diagnostic framework rather than just a critique
  • A four-bucket answer-contract taxonomy that predicts benchmark-level undercount magnitude from contract composition
  • Large-scale semantic audit of 37,033 official errors across six VLMs (InternVL3-8B, Qwen2.5-VL-7B, Gemma3-12B, Llama-3.2-11B, GPT-5.4, Claude Sonnet 4.6) and six benchmarks (ST-VQA, TextVQA, DocVQA, InfographicVQA, ChartQA-H/M)
  • Demonstration that benign prompt and context rewrites flip 1.8–29.4% of question-level correctness without changing the underlying task, with official score drops distinct from true score drops
  • A released CPU-only bidirectional contract repair pipeline that partially recovers the undercount

Results

  • ST-VQA undercount: 28–48% of official errors are semantically acceptable; mean judge correction +9.2 pp (open-weight), +8.7 pp (closed-source)
  • TextVQA: 21–32% of errors acceptable; mean +4.9 pp correction
  • DocVQA: 19–46% acceptable; mean +2.4 pp (partially absorbed by ANLS)
  • ChartQA-Machine: 0–4% acceptable; ~0 correction — consistent with 86.4% scalar contract composition
  • Contract-level false-negative rates: multi-span 39.2%, readout 36.4%, identifier-like 21.4%, scalar 21.1%
  • Benign flip rates (excluding Qwen2.5-VL outlier): ST-VQA 4.1%, TextVQA 5.5%, DocVQA 15.3%, InfoVQA 22.7%
  • Cross-judge replication: DeepSeek and Moonshot accept at 31.5% vs. 33.7% of errors respectively; per-dataset FN-rate alignment \(r = 0.999\)
  • Coarse open-weight leaderboard ordering is largely preserved (only single adjacent swaps), but score gaps lose interpretability when judge correction exceeds the gap itself

Limitations

  • The semantic judge is text-only and never sees the image; it assesses surface-form equivalence but cannot verify visual grounding
  • The four-bucket contract taxonomy is intentionally coarse and heuristic — does not claim to be a universal ontology of VQA answer types
  • Closed-source models (GPT-5.4, Claude Sonnet 4.6) are evaluated on stratified samples (500 or 200 items per benchmark) rather than full pools, limiting statistical power
  • Ranking inversion is minimal; the paper does not demonstrate that undercount changes model ordering conclusions at scale
  • Contract repair is described as only "partially recoverable" — full details of the CPU-only repair were truncated in the provided text

Relevance to Vision-Language Models

This paper is a direct validity audit of the standard evaluation infrastructure used to track VLM progress on text-rich understanding — ST-VQA, TextVQA, DocVQA, and related benchmarks are the primary yardsticks for OCR, document, and chart understanding in the VLM literature. The finding that mean judge corrections (+2.4 to +9.2 pp) routinely exceed adjacent-model score gaps (0.3–1.4 pp) implies that many published performance differentials between VLMs reflect evaluator contract differences as much as model quality differences. The answer-contract framework provides a principled diagnostic layer that complements official scores, and the perturbation sensitivity result (6–23% flip rates from benign prompt rewrites) directly challenges the assumption that a single benchmark number reliably characterizes a VLM's capability. Future VLM evaluations on short-answer benchmarks should accompany official scores with semantic audits and answer-type breakdowns to remain interpretable.