Skip to content

Sketch2Diagram: Generating Vector Diagrams from Hand-Drawn Sketches

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Sketch2Diagram introduces SKeTIkZ, a dataset of 3,231 hand-drawn sketch / TikZ code pairs, to benchmark the largely unexplored task of converting sketches to vector diagrams. Alongside the dataset, the authors develop ImgTikZ, a 6.7B-parameter VLM (DeepSeek Coder + SigLIP + MLP adapter) that matches GPT-4o in subjective alignment despite its compact size. Key enablers are two data augmentation strategies and a multi-candidate generation scheme with a diagram-specialized selector.

Problem

No public dataset pairs hand-drawn sketches with vector diagram code (TikZ), making it impossible to train or fairly evaluate sketch-to-diagram models. Existing work addresses text-to-diagram or image-to-diagram from clean rendered images, not noisy real-world sketches from paper, whiteboards, or tablets.

Method

Dataset (SKeTIkZ): TikZ codes from DaTikZ are rendered, filtered to five diagram categories (Tree, Graph, Architecture, Neural Networks, Venn), and paired with human-drawn sketches (28 annotators; paper/whiteboard/tablet). Result: 3,231 pairs (2,585 train / 323 val / 323 test).

Model (ImgTikZ): LLaVA 1.5 architecture with DeepSeek Coder 6.7B (code-specialized LLM) replacing the language model and SigLIP as the vision encoder; adapter is a two-layer MLP. Two-stage training: Stage 1 freezes LLM + encoder, trains only the adapter on arXiv figure captioning data; Stage 2 adds LoRA to the LLM and trains on TikZ-generation data.

Data augmentation: - AugTikZ (diagram variation): GPT-3.5 modifies TikZ source to create structurally varied diagrams (556K samples). - ImgAugTikZ (image variation): applies noise, background interference, brightness/contrast shifts, rotation, and distortion to clean rendered images to simulate sketch noise (714K samples).

Inference: - Iterative generation (ImgTikZ-IG): regenerates on compilation failure, up to M=5 attempts. - Multi-candidate generation (ImgTikZ-MCG): generates K=20 TikZ candidates, selects the best via D-SigLIP — a SigLIP model fine-tuned with a contrastive linear head to rank sketch–diagram similarity using cosine similarity between sketch image and rendered candidate image.

Key Contributions

  • SKeTIkZ dataset: 3,231 sketch/TikZ pairs from real-world tools (paper 78.8%, whiteboard 10.7%, tablet 10.5%); largest and most diverse such dataset.
  • ImgTikZ model: 6.7B open-source VLM achieving GPT-4o-level subjective alignment on sketch-to-diagram.
  • Two data augmentation strategies (AugTikZ + ImgAugTikZ) demonstrating substantial performance gains in ablation.
  • D-SigLIP selector for image-similarity-based candidate selection during inference; shown to outperform vanilla CLIP for this task.
  • Benchmarking of closed (GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet) and open-source (LLaVA-Next) VLMs, documenting systematic failures on this task.

Results

  • Compilation success rate (CSRavg): ImgTikZ-IG: 0.767, ImgTikZ-MCG: 0.799; vs. Claude 3.5: 0.544, GPT-4o: 0.479, GPT-4o-mini: 0.376, LLaVA-Next: 0.350.
  • Cumulative CSR (5 attempts): ImgTikZ-IG reaches ~100%; other models plateau at 0.8–0.9.
  • Image similarity (D-SigLIP): ImgTikZ-MCG: 0.821 (best); Claude 3.5: 0.753, GPT-4o: 0.695, LLaVA-Next: 0.315.
  • Subjective alignment (1–5): Claude 3.5: 3.32, ImgTikZ-MCG: 3.13, GPT-4o: 3.00; even best model at ~3.3 indicates only ~50–60% of diagram elements captured.
  • Subjective quality (1–5): Claude 3.5: 3.54 (best), ImgTikZ-MCG: 3.30, GPT-4o: 3.20.
  • Character similarity (CharSim): Claude 3.5: 0.671 (best); ImgTikZ-MCG: 0.594; GPT-4o: 0.611.
  • Ablation — augmentation: Removing ImgAugTikZ drops ImageSim from 0.734 to 0.668; removing both drops to 0.601.
  • Ablation — SKeTIkZ-only training: ImageSim drops from 0.734 to 0.513; CSRavg from 0.767 to 0.533.
  • Candidate scaling: Performance plateaus beyond K=5 with CLIP selector; D-SigLIP continues to improve up to K=20.
  • Correlation: Human alignment correlates most strongly with ImageSim (Pearson r=0.759), weakly with CodeSim (r=0.365).

Limitations

  • SKeTIkZ is restricted to TikZ output; no SVG, HTML, or other vector formats.
  • ImgTikZ significantly underperforms Claude 3.5 and GPT-4o on character recognition (CharSim), suggesting SigLIP resolution limits text legibility from sketches.
  • Paper and whiteboard sketches cause substantially larger performance drops than tablet sketches (up to −28% CharSim for whiteboards), and image augmentation only partially closes this gap.
  • Multi-candidate generation increases inference compute (K=20 renders); image-quality metrics are not incorporated into training, only at inference.
  • Hallucination risk during "information completion" (sketch details not explicit in the sketch) is acknowledged but unaddressed.
  • High-quality diagram generation remains unsolved: even the best model scores ~3.3/5 in alignment.

Relevance to Vision-Language Models

This work benchmarks frontier VLMs (GPT-4o, Claude 3.5 Sonnet, LLaVA-Next) on a challenging image-to-code generation task, exposing systematic weaknesses in spatial layout reasoning and character recognition from noisy visual inputs — capabilities assumed but rarely stress-tested in standard VLM evaluations. The ImgTikZ architecture and D-SigLIP selector demonstrate that a compact, domain-adapted open-source VLM with targeted augmentation and inference-time search can match closed-model performance, which is a general finding transferable to other image-to-code tasks (SVG, HTML). SKeTIkZ's compilation-based and image-similarity evaluation protocol offers a more grounded alternative to standard VLM benchmarks for assessing structured visual generation.