DaVinci: Reinforcing Visual-Structural Syntax in MLLMs for Generalized Scientific Diagram Parsing¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; DaVinci: RL-driven visual-structural syntax for scientific diagram parsing

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

DaVinci is a 7B-parameter MLLM that parses scientific diagram images into editable TikZ code via a two-stage pipeline: supervised fine-tuning on a curated TikZ-30K dataset, followed by GRPO-based reinforcement learning with a hybrid reward over code, vectorized representation, and rendered image. It achieves a 97.60% compile rate, surpassing GPT-5 and Claude-Sonnet-4 on the DAT IkZv3 benchmark while using far fewer parameters.

Problem¶

Existing MLLMs fail at scientific diagram-to-code parsing because the task demands pixel-level geometric precision, strict TikZ syntax adherence, and global structural planning over diverse visual primitives. Current SFT-only approaches are further hampered by noisy drawing orders in training data and reward signals that rely on error-prone OCR for text extraction.

Method¶

DaVinci uses a two-stage framework on top of Qwen2.5-VL-7B-Instruct:

Stage 1 — Cold-start SFT on TikZ-30K: 366K raw TikZ snippets are filtered to 225K high-quality samples (via Qwen-2.5-VL-32B quality scoring), then drawing order is normalized using Qwen3-Coder-480B to follow a semantics-guided constructive protocol. LLM-generated inline comments are injected as planning scaffolds. Stratified sampling yields 30K SFT samples (TikZ-30K).

Stage 2 — GRPO reinforcement learning: The policy generates $G=10$ candidate TikZ sequences per prompt. Advantage is estimated group-relatively: $$\hat{A}_k = \frac{R_{\text{hybrid}}^{(k)} - \text{mean}\{R_{\text{hybrid}}^{(j)}\}_{j=1}^G}{\text{std}\{R_{\text{hybrid}}^{(j)}\}_{j=1}^G}$$

The hybrid reward $R_{\text{hybrid}} = R_{\text{text}} + R_{\text{geom}} + R_{\text{img}} + R_{\text{pass}}$ combines: - $R_{\text{text}}$: Spatio-textual reward extracted error-free from PDF vectorization (via PyMuPDF), using exact-then-Levenshtein matching resolved by dIoU. - $R_{\text{geom}}$: Geometric primitive reward via Hungarian-algorithm bipartite matching on lines/rectangles/circles from PDF, scored by $\exp(-k \cdot C(e_p, e_g))$. - $R_{\text{img}}$: Combined DreamSim + normalized MSE image fidelity. - $R_{\text{pass}}$: Compile success (failed code gets minimum reward on all components).

Key Contributions¶

Two-stage SFT→RL framework explicitly separating visual primitive learning from structural relationship learning.
TikZ-30K dataset with drawing-order normalization and LLM-injected comment annotations as planning scaffolds.
Hybrid reward function leveraging PDF vectorization for extraction-error-free spatio-textual ($R_{\text{text}}$) and geometric ($R_{\text{geom}}$) reward signals, avoiding OCR errors.
Empirical finding that explicit chain-of-thought reasoning does not improve structured code generation; inline comments as implicit scaffolding outperform separate reasoning traces.

Results¶

DaVinci-7B compile rate: 97.60% vs. Claude-Sonnet-4-Thinking 86.90%, GPT-5-Default 72.88%, Gemini-2.5-Pro-Thinking 69.93%, best open-source (Qwen2.5-VL-72B) 80.04%.
DreamSim: 84.83 — best among open-source models; Gemini-2.5-Pro-Thinking scores 88.20 but with a 27.67% lower compile rate.
MSE: 61.81 — best across all models (proprietary and open-source).
SigLIP: 93.93 — best open-source; second overall behind Gemini-2.5-Pro-Thinking (95.59).
RL post-training raises compile rate from 84.50% (DaVinci-SFT-7B) to 97.60% while decreasing cBLEU (55.13→6.57 from SFT's 7.52), confirming visual fidelity and code similarity are decoupled.
DetikZify-V2-8B (specialized baseline): 78.60% compile, DSIM 82.63; DaVinci-7B improves compile by +19pp and DSIM by +2.2pp.
Evaluated on DAT IkZv3 test set (542 samples).

Limitations¶

Remaining compile failures concentrated on dense visualizations (e.g., scatter plots) where the model over-generates data points, exceeding context length.
RL training requires 8×H100-80G GPUs for 500 steps; the 28K RL split may limit generalization to diagram categories underrepresented in TikZ-30K.
The hybrid reward is specific to TikZ/PDF output; it does not transfer to SVG or other code formats without re-engineering the vectorization pipeline.
Human evaluation (BWS) results are truncated in the provided text; full human-eval numbers are not available to assess.
The finding that explicit reasoning does not help structured code generation is noted but left unexplored.

Relevance to Vision-Language Models¶

DaVinci is directly relevant to the image-to-structured-output capability of MLLMs, pushing beyond natural-language generation into precise, computable code. The RL post-training regime with multi-modal, extraction-error-free reward signals is a transferable recipe for any VLM task requiring geometric or spatial precision. The finding that a 7B model with targeted data curation and RL can surpass frontier proprietary models (GPT-5, Claude-Sonnet-4) on a structured generation benchmark challenges assumptions about scale being the primary driver of VLM capability. The decoupling of code-level similarity from visual fidelity also has implications for how VLM benchmarks in diagram/chart understanding should be designed.