Skip to content

Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation

🕒 Published (v1): 2025-01-06 18:57 UTC · Source: Arxiv · Venue: CVPR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AutoConverter is a multi-agent GPT-4o harness that automatically converts open-ended VQA questions into challenging multiple-choice format by orchestrating specialized proposer, reviewer, refiner, selector, and correctness-evaluator agents. It addresses critical reproducibility failures in existing open-ended VLM evaluation—rule-based metrics achieve only 0.09 Spearman correlation with true model rankings, and a single GPT-4o version bump inflates scores by 6%. Using AutoConverter, the authors construct VMCBench: 9,018 questions from 20 datasets evaluated across 33 state-of-the-art VLMs.

Problem

Open-ended VQA benchmarks are unreliable for VLM evaluation: rule-based metrics (exact match, BLEU, ROUGE) achieve Spearman \(\rho = 0.09\) vs. true model performance on VQAv2 and misclassify ~66% of GPT-4o's correct answers as wrong, while model-based evaluation (GPT-4o-as-judge) shifts absolute scores by 6% across version updates (0513→0806 on MMVet) despite preserving ranking correlation (\(\rho = 1.0\)). Multiple-choice evaluation avoids both problems but is expensive to construct—particularly the distractor generation step.

Method

AutoConverter is a GPT-4o-powered multi-agent pipeline with two parallel objectives:

Increasing difficulty: Five specialized proposer agents (one per error type—concept misunderstanding, visual misinterpretation, reasoning error, data processing error, question bias) generate candidate distractors with rationales from the image-question-answer triple \((v, q, a)\). A reviewer agent iteratively critiques plausibility and challenge level, the proposers refine accordingly, and a selector agent picks the \(N=3\) most difficult distractors from the pool.

Ensuring correctness: A correctness evaluator agent scores each generated question on a 5-point Likert scale (5 = single unambiguous correct answer). Questions scoring below 4 are sent to a refiner agent that adjusts distractors based on evaluator feedback; this loop runs for up to 3 iterations. Only questions achieving score ≥ 4 are retained; uncertain cases undergo manual human verification.

VMCBench is assembled by applying AutoConverter to 12 open-ended datasets (conversion) and 8 existing multiple-choice datasets (distractor refinement), totaling 9,018 questions.

Key Contributions

  • AutoConverter: a multi-agent harness for automated, quality-controlled MCQ generation from open-ended VQA, requiring no external knowledge graphs or supervised training
  • Systematic evaluation failure analysis: quantified rule-based (\(\rho=0.09\)) and model-based (6% absolute shift per GPT-4o version) evaluation pathologies across 12 VLMs on VQAv2 and MMVet
  • VMCBench: 9,018 unified MCQ questions aggregated from 20 existing VQA datasets (general, reasoning, OCR, doc/chart), evaluated on 33 VLMs
  • Ablation study demonstrating each agent component's contribution to correctness and difficulty on MMMU

Results

  • AutoConverter-generated distractors match or exceed human-crafted difficulty: VLMs achieve similar or lower accuracy on AutoConverter questions vs. originals across MMMU, MathVista, and AI2D
  • Human annotation (correctness score = 5 filter) shows only 3% of questions are incorrect; of those, 52% stem from errors in the original dataset answers, not AutoConverter
  • Correctness evaluator calibration: 95% of score-5 questions are human-verified correct (vs. 51% at score 1)
  • Ablation on MMMU (average model accuracy, lower = harder):
  • Full AutoConverter: 40.7% avg; Human-created: 41.8%; Naive conversion: 46.0%
  • Removing refiner: 37.8% correctness score drop (−8.7% relative), accuracy 37.8% avg
  • Removing reviewer: avg accuracy rises to 42.7% (difficulty loss)
  • Removing any single proposer type: avg accuracy rises 1.6–3.5% relative
  • Rule-based eval reports GPT-4o at 32% on VQAv2; human-corrected accuracy is 88%; model-based proxy achieves 0.95 human agreement

Limitations

  • AutoConverter relies entirely on GPT-4o as backbone for all agents; cost scales with number of questions and refinement rounds (up to 3 rounds × multiple agents per question)
  • GPT-4o version changes may affect AutoConverter's own outputs, creating the same reproducibility risk it aims to resolve in evaluators
  • Correctness evaluator itself achieves only 95% precision at score 5—residual 5% incorrect questions require human review
  • VMCBench fixes the answer set at benchmark construction time; adversarial distractors may become easier as VLMs improve, requiring periodic regeneration
  • \(N=3\) distractors chosen for common convention; no analysis of optimal distractor count

Relevance to Harnesses / Meta-Harnesses

AutoConverter is a concrete multi-agent harness for benchmark construction automation: it composes specialized role-playing agents (proposer, reviewer, refiner, selector, evaluator) in an iterative pipeline governed by a scalar quality signal (Likert correctness score), demonstrating the harness-as-quality-control-loop pattern. For someone tracking meta-harnesses, this paper is a direct example of an evaluation-pipeline harness—a harness whose output is not task completions but validated test data, a level of abstraction above standard task-solving agents. The iterative refinement loop (score < threshold → refine → re-evaluate) mirrors self-reflection and self-repair patterns central to robust agentic harness design. VMCBench also provides a reproducible evaluation substrate that downstream harness evaluations can use as a controlled, version-stable benchmark.