EduArt: An educational-level benchmark for evaluating art history knowledge in large language models

🕒 Published (v1): 2026-07-02 10:43 UTC · Source: Arxiv · link

Why this paper was selected

Domain-specific benchmark for art history in LLMs; benchmark methodology over synthetic baselines

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EduArt is an 871-item benchmark of human-authored art history questions drawn from Italian secondary-school exercises and US AP Art History exams, covering seven question formats and evaluated across 12 multimodal LLMs under two prompting conditions. Multiple-choice accuracy saturates near ceiling for frontier models (\(\geq\)94%), masking catastrophic format-dependent failures: the same models drop to 23.9% on open completion and 6.2% on error identification. The benchmark exposes that art-historical knowledge and its deployable application are empirically dissociable capabilities.

Problem

General-purpose benchmarks sacrifice diagnostic depth within single disciplines, and existing art-focused VQA benchmarks rely on synthetically generated questions without reporting item-level psychometric properties. This makes it impossible to disentangle genuine knowledge gaps from format sensitivity or poorly constructed items—especially for a field like art history that inherently requires joint visual and linguistic reasoning.

Method

Questions were sourced from two real-world educational corpora: MyZanichelli (668 Italian-language questions across six open/closed formats) and College Board AP Art History exams 2013–2024 (203 English-language MCQs). Questions were extracted via Playwright browser automation and dual-model cross-validation (Gemini 3 Flash + Anthropic Sonnet 4.5) for the Italian source, and programmatic PDF parsing for the AP source. All items were normalized to a canonical JSON schema including image paths, language codes, and answer keys.

Each of 12 models (GPT-5.5, GPT-5.4 Mini/Nano, Gemini 3.1 Pro/Flash Lite/3.5 Flash, Claude Opus/Sonnet 4.6, Haiku 4.5, Qwen3-VL-235B, Mistral Large 3 675B, Pixtral Large) was evaluated under (1) a default answer-only condition and (2) a motivation condition requiring a 2–4 sentence written justification grounded in visual evidence. Accuracy metrics are format-stratified: exact match for MCQ radio, F1 for multi-select and error identification, statement/element/blank accuracy for true-false, positioning, and completion tasks respectively. A logistic regression was fitted to isolate independent effects of format, language, image presence, and model identity. Benchmark quality was characterized via Classical Test Theory (CTT): per-item difficulty (proportion correct) and discrimination (point-biserial correlation with total score).

Key Contributions

• First art history benchmark built entirely from human-authored educational materials (not synthetically generated), providing verifiable ground truth at educational level.
• Seven question formats spanning recognition to production tasks, enabling direct format-controlled comparisons within the same domain.
• Full psychometric characterization via CTT: mean discrimination 0.514; 82.3% of items classified as good discriminators.
• Empirical demonstration that question format is a strong independent predictor of accuracy, causing frontier models to diverge by up to 70+ percentage points between MCQ and open/error formats.
• Paired default vs. motivation condition analysis revealing that elicited chain-of-thought reasoning predominantly reduces accuracy in a model-family-dependent pattern.
• Bilingual (Italian/English) design enabling language-effect estimation under matched format conditions.

Results

• Macro-average accuracy (across 7 formats) ranges from 29.1% (Pixtral Large) to 82.8% (Gemini 3.1 Pro Preview).
• MCQ exact-match: six models exceed 90%; top five (Gemini 3.1 Pro 95.7%, Gemini 3.5 Flash 96.5%, GPT-5.5 95.1%, Claude Opus 4.6 94.6%, Claude Sonnet 4.6 94.3%) have overlapping 95% CIs—MCQ alone cannot distinguish frontier models.
• Open completion (completion_open): Claude Opus 4.6 falls to 23.9%.
• Error identification (select_errors): Claude Sonnet 4.6 falls to 6.2%.
• Psychometric quality: mean item discrimination 0.514; 82.3% of items are good discriminators.
• Motivation condition changes accuracy predominantly negatively, with the direction and magnitude being model-family-dependent.

Limitations

• Questions are under copyright and cannot be redistributed; only the evaluation pipeline code is public, limiting direct dataset reproducibility.
• Images are low-to-medium resolution (mean ~470×463 px, ~213 KB); results on image-dependent items are a lower bound of what higher-fidelity inputs could achieve.
• Non-MCQ formats are exclusively Italian, so language and format effects cannot be fully orthogonalized for those types.
• Temperature was not held constant across providers (0 for Anthropic/open-weight, 1 for Gemini/GPT-5.x), introducing a confound for inter-family comparisons.
• Coverage is restricted to Italian Renaissance painting (Zanichelli) and the AP Art History curriculum; generalization to other art-historical periods or non-Western traditions is untested.

Relevance to Vision-Language Models

EduArt directly probes the limits of multimodal VLMs by constructing tasks where language shortcuts are insufficient: open completion and error-identification formats require producing or manipulating content rather than selecting from fixed options, exposing the gap between pattern-matched recognition and genuine visual–linguistic reasoning. The finding that models exceeding 94% MCQ accuracy can collapse below 10% on error identification reinforces the MathVerse-style concern that high VQA scores may reflect language prior exploitation rather than true visual grounding. The motivation condition further shows that chain-of-thought elicitation—a standard VLM inference-time technique—can actively degrade performance, with model-family-dependent effects that cannot be predicted from MCQ scores alone. For researchers tracking VLMs, EduArt provides a methodologically rigorous, psychometrically validated template for building domain-specific benchmarks that distinguish apparent from genuine multimodal reasoning.