Skip to content

DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DynaMath is a dynamic visual math benchmark of 501 seed questions, each encoded as a Python program that programmatically generates diverse concrete variants (numerical values, graph structures, geometric transformations, etc.). Evaluating 14 VLMs on 5,010 generated questions reveals that worst-case accuracy is consistently ≤50% of average-case accuracy across all models, exposing systematic fragility in mathematical reasoning that does not afflict humans.

Problem

Existing visual math benchmarks (MATHVISTA, MATHVERSE, MATH-V) are static: each question has exactly one instance, so models can exploit memorized patterns to inflate scores and genuine reasoning robustness cannot be measured. There is no prior benchmark that tests whether VLMs generalize a solution procedure across semantically equivalent but visually/numerically varied instances.

Method

Each of the 501 seed questions is implemented as a hand-crafted Python program with a parameterized sampling space and a drawing function (Matplotlib/Pyglet) that synthesizes the visual context. Running the program with different sampled parameters yields arbitrarily many concrete question–image pairs with algorithmically computed ground-truth answers. During evaluation, 10 variants per seed (5,010 total) are generated. Variation types include: (1) numerical value substitutions, (2) geometric transformations (shape/angle/position), (3) function type variants (linear→quadratic etc.), (4) color variants, (5) symbolic substitutions (operator changes), (6) graph structure variants (topology/layout), and (7) real-life context variants (calendars, poker). Three metrics are reported: average-case accuracy \(A_\text{avg}\) (mean over all variants), worst-case accuracy \(A_\text{wst}\) (fraction of seeds answered correctly on every variant), and reasoning robustness \(\text{RR} = A_\text{wst}/A_\text{avg}\). Repetition consistency (RC, 5 repeated generations of the same variant) controls for output randomness.

Key Contributions

  • First systematic study of VLM robustness on visual mathematical reasoning via dynamically generated benchmarks.
  • DynaMath: 501 programs spanning 9 topics (Plane/Solid/Analytic Geometry, Algebra, Arithmetic, Puzzle Tests, Graph Theory, Statistics, Scientific Figures) and three difficulty levels (elementary, high school, undergraduate).
  • Formal definitions of worst-case accuracy, reasoning robustness (RR), and repetition consistency (RC) tailored for robustness evaluation.
  • Identification of the memorization phenomenon: models output the same answer (e.g., 2Ï€ for period) regardless of the visual conditions shown.
  • Large-scale evaluation of 14 closed- and open-source VLMs plus a 20-person human baseline.

Results

  • Human baseline: 77.3% average accuracy, stable across difficulty levels.
  • Best closed-source (zero-shot): Claude-3.5 Sonnet — 64.8% avg / 35.3% worst-case; GPT-4o — 63.7% / 34.7%; Gemini Pro 1.5 — 60.5% / 26.9%.
  • 3-shot CoT does not consistently help: Claude-3.5 drops from 64.8% → 62.5%; Gemini drops from 60.5% → 58.7%.
  • Best open-source: Qwen2-VL-72B — 55.1% avg / 28.3% worst-case; InternVL2-76B — 54.0% / 24.6%.
  • Worst open-source: LLaVA-v1.5-7B — 16.6% avg / 1.8% worst-case.
  • Scaling trend: Qwen2-VL 7B→72B improves avg from 42.1%→55.1%; InternVL2 8B→76B from 39.7%→54.0%.
  • Repetition consistency (RC): 92–99% across models — failures are not random; they are systematic on specific variants.
  • Consistent failure rate: 21.8% of seed questions cause consistent failure in GPT-4o; 29.9% in Qwen2-VL-72B.
  • Error breakdown (Claude-3.5 Sonnet): figure-reading 33.1%, reasoning 26.6%, calculation 18.9%, hallucination 17.8%, knowledge 3.6%.

Limitations

  • Only 10 variants per seed are evaluated; worst-case over 10 is a lower bound on true worst-case over the full combinatorial space.
  • Human baseline uses only 2 variants per seed (1,002 questions), so human robustness is not measured in the same framework as model robustness.
  • 31 of 501 programs use fixed images and randomize only text, limiting the scope of visual variation for those questions.
  • Programmatic generation restricts inclusion to questions with non-excessively-complex images, biasing toward structured/synthetic visuals over natural diagrams.
  • Benchmark is not designed as competition-level challenge; hardest frontier models (e.g., o1-series) are not evaluated.

Relevance to Vision-Language Models

DynaMath directly probes a known blind spot of VLM evaluation: static benchmarks conflate memorization with reasoning, and DynaMath's dynamic generation protocol cleanly separates the two. The worst-case / reasoning-robustness framing complements average-accuracy leaderboards and reveals that even the best VLMs (Claude-3.5, GPT-4o) fail more than half their seed questions on at least one variant—a gap that directly motivates research in robust visual encoding and compositional generalization. The memorization phenomenon and consistent-failure analysis provide diagnostic targets for future VLM training, including curriculum design and data augmentation strategies for visual math.