Exploring Agentic Workflows for Generating High Quality Math Visual Aids

🕒 Published (v1): 2026-07-10 17:12 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes an agentic, self-improvement pipeline for generating K-12 mathematical diagrams in TikZ, where an LLM generates QA question-answer pairs to evaluate each diagram, a VLM/LLM evaluates the diagram against those questions, and unsatisfied questions feed back into diagram regeneration. Human quality scores improve modestly (3.4 → 3.7 over two regeneration rounds), though the system remains limited by incomplete QA coverage and weak spatial reasoning.

Problem

State-of-the-art LLMs achieve only 73.9% accuracy generating K-12 math diagrams from detailed descriptions, and no robust automated evaluation mechanism exists to assess whether generated diagrams faithfully represent intended pedagogical content — blocking the construction of reliable self-improving generation loops.

Method

A four-stage pipeline operating on TikZ code and rendered images: 1. Diagram Generator: Produces an initial TikZ diagram from an enhanced image description. 2. QA Question Generator: An LLM produces four open-ended QA pairs (two accuracy, two clarity) with gold-standard answers, evaluated on the TikZ code rather than (or in addition to) the rendered image. 3. QA Evaluator + Verifier: A combined code+image representation is fed to a VLM/LLM with chain-of-thought prompting; a Verifier compares answers against gold standards to produce binary satisfied/unsatisfied signals. 4. Feedback Extractor + Regeneration: Unsatisfied QA answers are converted to textual feedback and injected into the diagram generator; the loop continues until all questions are satisfied or a maximum cycle count is reached.

Key design decisions: switched from yes/no questions (Cohen's \(\kappa = 0.11\)) to open-ended questions; switched from image-only evaluation to combined code+image evaluation to handle counting and spatial relationships; added chain-of-thought prompting to the evaluator.

Key Contributions

• An end-to-end agentic harness for iterative TikZ diagram generation, combining LLM-based QA generation with VLM-based evaluation and feedback-driven regeneration.
• Empirical evidence that combined code+image evaluation (97% human agreement) substantially outperforms diagram-only (68%) or code-only (81%) evaluation.
• Demonstration that open-ended QA questions are more reliable evaluators of visual diagram quality than yes/no questions for VLM-based assessment.
• Error taxonomy identifying two primary failure modes: incomplete QA coverage of diagram features and VLM limitations in spatial reasoning and object counting.

Results

• Human agreement with QA Evaluator: 68% (diagram-only), 81% (TikZ code-only), 97% (code + diagram).
• Diagram quality (1–5 human score): 3.4 (initial) → 3.65 (after regeneration 1) → 3.70 (after regeneration 2).
• QA unsatisfied rate: 24.3% (initial) → 18.3% (round 1) → 15.9% (round 2).
• 30% of diagrams were accepted after initial generation; the remaining 70% all required at least two regeneration rounds, suggesting the one-shot feedback loop rarely resolves hard cases.
• No controlled baseline comparison; results are preliminary descriptive findings (exact model configs and sample counts not preserved).

Limitations

• No reproducible benchmark: model configurations, prompt templates, and evaluation sample counts were not systematically recorded.
• QA generation lacks full coverage — questions miss critical diagram attributes (e.g., label values vs. label positioning), allowing incorrect diagrams to pass evaluation.
• VLM spatial reasoning is unreliable for counting elements, verifying distances, and assessing proportional layout.
• Regeneration loop stalls: 70% of diagrams still fail after round 1 and again after round 2, indicating feedback is often insufficiently actionable for TikZ code synthesis.
• Quality improvement is modest (3.4 → 3.7 on a 5-point scale) and may not meet deployment thresholds for K-12 use.
• Pipeline relies on TikZ as the diagram representation, limiting applicability to diagram types well-supported by TikZ.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct instantiation of a self-improvement meta-harness: a controller loop that wraps a generation component with automated evaluation and feedback extraction stages, routing outputs back into the generator until a quality criterion is met or a budget is exhausted. The explicit cycle termination condition (all QA satisfied OR max cycles) mirrors the loop-until-convergence pattern central to meta-harness design. The paper's finding that combining code and image representations at the evaluation stage substantially improves harness reliability is directly actionable for harness designers choosing what artifacts to expose to verifier agents. The failure analysis also surfaces a known meta-harness failure mode: when the quality signal (QA questions) is itself incomplete or miscalibrated, the feedback loop provides false convergence rather than genuine quality improvement.