Dysca: A Dynamic and Scalable Benchmark for Evaluating Perception Ability of LVLMs

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Dysca is a synthetic benchmark for evaluating Large Vision-Language Models (LVLMs) that generates novel images via Stable Diffusion and paired QA via rule-based templates, covering 20 perceptual subtasks across 51 image styles and 4 test scenarios. By generating both images and questions from scratch, it eliminates data leakage from publicly known datasets while enabling scalable, fine-grained perception evaluation. Evaluation of 26 LVLMs reveals consistent weaknesses in adversarial/print-attack robustness and sensitivity to question-type framing.

Problem

Existing LVLM benchmarks select images from public datasets, creating data leakage risk—prior work has confirmed these images enter LVLM training data. Additionally, benchmarks largely test only realistic images under clean conditions, leaving multi-stylized images and noisy scenarios (corruption, print attacks, adversarial attacks) unexplored, providing incomplete coverage of real-world perceptual failure modes.

Method

Dysca structures data as a 4-tuple (Metadata M, Prompt P, Image I, QA Q). Metadata decouples into four components—foreground, attribute, style, background—combined via pre-defined templates to generate text prompts. Stable Diffusion XL synthesizes images from these prompts; for OCR subtasks, TextDiffusion2 is used. Since all image content is known from the prompt, QA pairs are constructed rule-based (no annotation model needed): multi-choice (4-way), true-or-false, and free-form (image captioning against the prompt). Images are post-processed into 4 scenarios: clean, image corruption (MMCBench methods), print attack (deceptive wrong-option text overlaid), and PGD adversarial attack (using InstructBLIP as white-box proxy). Data cleaning uses CLIP-L-14 (text-image similarity threshold 0.75) and PP-OCRv3 (for OCR failure filtering), plus a top-6 model consensus pass; ~40% of generated samples are discarded. Two novel sensitivity metrics are introduced: SQ (sensitivity to question type, variance between normalized true-or-false and multi-choice scores) and SC (sensitivity to covariate/style shift, variance across 51 styles).

Key Contributions

• Dysca benchmark: 617K QA pairs (expandable, theoretically unlimited) spanning 20 perceptual subtasks and 51 image styles, fully synthesized to prevent data leakage.
• End-to-end generative pipeline: Stable Diffusion image generation + rule-based QA, removing hallucination-prone LLM annotation.
• Multi-scenario, multi-question-type coverage: 4 scenarios (clean, corruption, print attack, adversarial attack) × 3 question types (multi-choice, true-or-false, free-form).
• Two novel diagnostic metrics: SQ (question-type sensitivity) and SC (covariate/style-shift sensitivity).
• Validity demonstration: Spearman ρ ≥ 0.70 with MMBench and ρ = 0.90 (1.00 on realistic subset) with OCRBench, confirming synthetic evaluation correlates with real benchmarks.

Results

• GLM-4V achieves the highest clean-scenario score (82.09), followed by MiniCPM-V2.5 (78.75) and Gemini-1.5-Pro (77.79); no model exceeds 85.
• All models are near-random on blind (text-only) queries (~35%), confirming leakage mitigation.
• Corruption scenario: all models show <1% performance variation vs. clean—high robustness.
• Print attack: severe drops; GPT-4o −25.8% (75.69→56.10 inferred; table shows 56.34), Gemini-1.5-Pro −21.5% (77.79→61.05), GLM-4V −36.5% (82.09→52.09); mPLUG-Owl-2 most robust (−1.7%).
• Adversarial attack: EVA-CLIP–based models collapse; XComposer-VL −57.6% (71.40→30.28, below random); models with different encoders drop 1–5%.
• Spearman ρ with MMBench: 0.70 (all styles), τ = 0.60; with OCRBench: ρ = 0.90 (1.00 on realistic), τ = 0.80 (1.00); SeedBench-2 realistic: ρ = 0.64, τ = 0.62.
• LVLMs perform better on landmark and object recognition; style and movie recognition are hardest.
• Commercial models (GPT-4o, Gemini) refuse or underperform on race/age/gender tasks due to safety training.

Limitations

• Synthetic images may not fully represent real-world distribution; correlation with SeedBench-2 is weaker (ρ = 0.46 all-style) until filtered to realistic-style images only.
• Adversarial attacks use InstructBLIP as a single white-box proxy; transferability findings may not generalize to all architectures.
• Free-form evaluation uses SentenceTransformer similarity against the generating prompt—an indirect and potentially imprecise measure of captioning quality.
• Style list excludes artist-name styles to avoid copyright concerns, limiting certain artistic perception evaluations.
• Synthesis quality depends on T2I model capability; ~40% sample rejection rate indicates non-trivial generation failure.

Relevance to Vision-Language Models

Dysca directly addresses a critical methodological gap in LVLM evaluation: data leakage from benchmark contamination, which inflates apparent capability. For researchers tracking VLMs, it provides a leakage-free, fine-grained probe of perceptual abilities across diverse visual distributions—revealing that even top models fail badly under print attacks and adversarial perturbations. The paper's two new metrics (SQ and SC) introduce a principled way to assess model consistency across question formats and image styles, which is practically important for deployment. The finding that language model quality dominates visual encoder quality across scales informs architecture design decisions in the VLM community.