SciDiagramEdit: Learning to Edit Scientific Diagrams from Paper Revisions¶

🕒 Published (v1): 2026-07-16 17:58 UTC · Source: Arxiv · link

Why this paper was selected

Figure editing via paper revision diffs; novel multimodal grounding for document agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

SciDiagramEdit is a benchmark and skill-evolution framework for instruction-driven editing of scientific figures. It mines 364 before/after figure pairs from arXiv revision histories and trains a self-improving SVG-editing agent whose skill specification is iteratively refined from execution traces and author demonstrations. The agent achieves a semantic checklist success rate of 0.932, surpassing all single-pass baselines and competitive raster regenerators.

Problem¶

Existing tools address figure generation from scratch; none supports targeted editing of an existing scientific figure given a natural-language instruction. Raster editors (GPT-Image) re-render the whole image in one pass, exposing no addressable vector primitives for local operations such as relabeling a component or re-routing arrows. Single-pass LLM editors are brittle across the long tail of edit types because the space of editing instructions is too diverse for any fixed scaffold to anticipate.

Method¶

Figures are represented as \(F = (V, A)\), where \(V\) is the SVG source (vectorised schematic primitives) and \(A = \{a_k\}_{k=1}^m\) is the set of embedded raster panels retained from the original. Each sample is annotated with a per-sample checklist \(Q = \{(q_j, C_j, a^*_j)\}_{j=1}^n\) of atomic binary or multi-choice questions grounding the instruction.

A three-component training loop drives skill evolution:

Editor \(\mathcal{E}\) — an agentic code-writing subprocess with file-system and Python tools; it reads a learnable skill specification \(S\) (a SKILL.md plus optional workflows/ and tools/ files) verbatim into its system prompt and produces an edited figure \(F^{\text{out}}\) plus execution trace \(\tau\).
Judge \(\mathcal{J}\) — scores each edit with a composite \(r = r_{\text{aes}} \cdot r_{\text{sem}}\). Semantic faithfulness \(r_{\text{sem}}\) is checklist verification accuracy; aesthetic preference \(r_{\text{aes}} = \mathbb{1}[F^{\text{out}} \succeq F^{\text{ref}}]\) is a pairwise VLM comparison against the author's revision to suppress single-image scoring noise.
Coach \(\mathcal{C}\) — consumes traces, judge scores, and the author's reference figure \(F^{\text{ref}}\) as a demonstration, then emits a file-level patch \(P = \mathcal{C}(S, \mathcal{B}, \tau, r, H)\) to update \(S\). A top-\(K\) frontier tracks the best-scoring skill snapshots; the optimal skill is \(S^* = \arg\max_S \bar{r}(S; D_{\text{val}})\).

Skill evolution runs for two epochs on \(D_{\text{train}}\), with validation gating to reject regressions.

Key Contributions¶

SciDiagramEdit benchmark: 364 before/after figure pairs mined from arXiv version histories across 23 primary subjects, annotated with 2,628 atomic editing claims capturing genuine author revision intent.
Skill-evolution training loop: an Editor–Judge–Coach cycle that distils author demonstrations and execution traces into a portable, learnable skill specification rather than a hand-engineered scaffold.
Composite judge: gating semantic faithfulness on aesthetic preference (\(r = r_{\text{aes}} \cdot r_{\text{sem}}\)) rather than averaging, preventing credit for edits that are faithful but visually degraded.
Transferable skills: the skill specification evolved on one backbone (GPT-5.5) transfers to weaker backbones with comparable gains, suggesting learned rules are generalisable rather than model-specific.

Results¶

Semantic success rate: Ours 0.932 vs. GPT-Image-2 0.882, AutoFigure-Edit (GPT-5.5) 0.844.
Instruction-following win rate (pairwise vs. author revision): Ours 0.756 vs. GPT-Image-2 0.778, AutoFigure-Edit (GPT-5.5) 0.573.
Aesthetic IAA / ISTA: Ours 47.93 / 45.58 vs. Target 48.25 / 46.49; Ours is the strongest vector-aware method but trails raster regenerators (GPT-Image-2: 49.60 / 45.78).
Skill ablation across backbones: evolved skill yields consistent \(\Delta\) semantic gains (+0.011 to +0.054) and \(\Delta\) aesthetic gains (+0.037 to +0.076) over no-skill baselines across GPT-5.1 through GPT-5.5.
User study (30 instances, 5 participants): Ours wins 0.59 vs. GPT-Image-2 and 0.63 vs. AutoFigure-Edit (GPT-5.5) on semantic; 0.54 and 0.68 respectively on aesthetic.
Validation curve: best-so-far skill score rises monotonically over 10 evolution iterations.

Limitations¶

Trails closed-source raster regenerators (GPT-Image-1.5, GPT-Image-2) on blind pairwise aesthetic win rate, as raster regenerators can leverage global re-rendering to polish visual appearance.
Benchmark skews heavily toward ML/CS papers: CS.LG, CS.CL, CS.CV, CS.RO jointly account for 73.6% of samples, limiting coverage of non-ML scientific styles.
Publication-year distribution skews toward 2023–2024 (47%), so revision patterns from older or future works may differ.
Evaluation relies on a VLM judge whose pairwise aesthetic preference may not fully capture domain-specific scientific-diagram conventions.

Relevance to Agentic AI / LLM Agents¶

SciDiagramEdit is a direct instantiation of the self-improving agentic loop paradigm — Editor, Judge, and Coach — where the unit of evolution is a portable skill specification rather than model weights, placing it in the same lineage as VOYAGER, EvoSkill, and Trace2Skill. The demonstration-aware Coach, which uses author-drawn revision targets as behavioral demonstrations to diagnose systematic gaps in the Editor's trace, is a concrete mechanism for grounding skill evolution in external ground truth rather than self-assessed reward alone. The top-\(K\) frontier with validation gating offers a practical recipe for preventing skill regression during open-ended iterative refinement. For researchers tracking LLM agents, this paper shows how agentic skill evolution scales to a structured multimodal domain (SVG editing) where execution feedback is rich and verifiable.