Skip to content

VCode: a Multimodal Coding Benchmark with SVG as Symbolic Visual Representation

🕒 Published (v1): 2025-11-04 18:00 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VCode is a multimodal benchmark that reframes image understanding as SVG code generation: given a natural image, a model must produce executable SVG that preserves symbolic meaning, evaluated via a rendered-image VQA protocol (CodeVQA). The companion VCoder framework wraps any base VLM with an iterative revision loop and external vision-tool calls, yielding +12.3 overall points over the strongest baseline (Claude-4-Opus, 41.7 → 54.0).

Problem

Existing coding benchmarks are linguistically centric (HumanEval, SWE-Bench), while multimodal benchmarks evaluate text-answer generation from pixel images. No benchmark tests whether a model can encode a natural image into compact, executable, symbolically faithful code; and no standard agentic harness exists to augment VLMs with iterative visual feedback for this task.

Method

Benchmark (VCode). 464 image-question pairs drawn from MM-Vet (218), MMMU-dev (146), and CV-Bench (100). A model receives only a raw image and must output SVG code; the SVG is rendered and a policy VLM (GPT-4o-mini) answers original questions over the render. Correctness = symbolic preservation. Two automatic metrics: SigLIP cosine similarity between original and rendered embeddings, and CodeVQA accuracy.

Agentic harness (VCoder). Two orthogonal augmentations applied at test time to any base coder:

  1. Thinking with Revision — iterative loop: (a) render current SVG \(C^{(t)}\), (b) compute discrepancy signal \(\Delta^{(t)} \leftarrow \psi(V, \tilde{V}^{(t)})\) by prompting the VLM to articulate spatial/color/structural differences, (c) regenerate \(C^{(t+1)}\) conditioned on \((C^{(t)}, \tilde{V}^{(t)}, \Delta^{(t)})\); repeat for \(T\) iterations.

  2. Acting with Visual Tools — equips the coder with external perception modules: Florence-2 for bounding boxes, SAM-2 for segmentation polygons (adaptively simplified), OpenOCR for text, and an object detector for semantic labels. Outputs are translated into structured SVG hints injected as coding context.

Key Contributions

  • VCode benchmark: first benchmark requiring SVG code generation from natural images across general, professional, and visual-centric domains.
  • CodeVQA evaluation protocol: measures symbolic fidelity through render-then-VQA rather than pixel similarity alone.
  • VCoder: a test-time agentic harness combining iterative self-revision with external tool-augmented perception.
  • Empirical analysis of 27+ frontier VLMs revealing consistent failure modes (3D depth, professional knowledge, irregular object boundaries).

Results

  • VCoder (on Claude-4-Opus): Overall 54.0 vs. 41.7 baseline (+12.3 pts); MM-Vet 54.2 (+16.7), MMMU 48.6 (+6.2), CV-Bench 61.3 (+11.4).
  • Best frontier baseline: GPT-5 at 46.8 Overall, SigLIP 72.3.
  • Raw-image upper bound (GPT-4o-mini on original images): 61.7 Overall — even VCoder does not close this gap.
  • SVG specialists (OmniSVG, StarVector-8B) score near 0 on long natural-image SVG, exposing domain mismatch.
  • Short SVG outputs (Qwen-2.5-VL: 0.3K tokens) correlate strongly with lower performance; token length correlates with expressiveness.
  • Human studies confirm both humans and VLMs score lower on rendered SVGs than originals, with consistent relative rankings.

Limitations

  • VCoder's revision loop increases inference cost proportionally to \(T\) iterations; the paper does not report wall-clock or token-cost overhead.
  • The benchmark is small (464 pairs); professional-knowledge subset is 146 pairs.
  • CodeVQA correctness depends on the policy model (GPT-4o-mini), introducing evaluator bias.
  • VCoder is only demonstrated on Claude-4-Opus; generalization to other base coders is not systematically ablated.
  • 3D perception and depth-order tasks remain near chance even after VCoder augmentation.

Relevance to Harnesses / Meta-Harnesses

VCoder is a concrete instantiation of a test-time augmentation harness: it wraps an arbitrary base VLM with a structured loop (detect → critique → revise) and a tool-dispatch layer (detector, segmenter, OCR), without retraining the model. This mirrors the meta-harness pattern of orchestrating multiple specialized modules around a frozen foundation model. The iterative revision loop (\(C^{(t)} \to \tilde{V}^{(t)} \to \Delta^{(t)} \to C^{(t+1)}\)) is a domain-specific instantiation of the reflect-and-refine scaffolding common in coding and reasoning harnesses. The modular tool-dispatch design — where each tool outputs structured, code-compatible cues — is directly analogous to harness designs that convert unstructured perception into actionable signals for a reasoning core.