Skip to content

Rendering-Aware Reinforcement Learning for Vector Graphics Generation

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RLRF (Reinforcement Learning from Rendering Feedback) addresses the non-differentiable rendering bottleneck in autoregressive VLM-based SVG generation by using rendered rollout outputs as reward signals for GRPO-based RL post-training. Applied after supervised fine-tuning, RLRF achieves state-of-the-art SVG fidelity and code compactness, substantially outperforming both SFT-only baselines and closed-source frontier models like GPT-4o.

Problem

Autoregressive VLMs trained for SVG code generation (e.g., StarVector) use token-level cross-entropy loss and never observe the rendered visual output during training. This causes hallucination, looping, and poor out-of-distribution generalization. Differentiable rasterizers (DiffVG) cannot be applied because discrete token sampling breaks gradient flow through the generation path.

Method

Two-stage training: 1. SVG-SFT: Standard supervised fine-tuning on 1.7M image-SVG pairs (SVG-Stack) with next-token prediction to establish SVG fluency. 2. RLRF: Post-training via GRPO on 20k high-entropy SVG-Stack samples. For each input image, G=64 SVG rollouts are sampled, rendered with CairoSVG, and compared to the input to compute a composite reward: - Rimg (L2): Normalized pixel-level L2 distance, clipped to [−1,1]. - Rimg (L2-Canny): Same but on Canny edge maps (with dilation + Gaussian blur). - Rsim (DreamSim): Cosine similarity via concatenated CLIP/OpenCLIP/DINOv2 ViT-B/16 features; DreamSim-Canny variant for shape focus. - Rlen (SVG length): Quadratic penalty when predicted token count exceeds 1.5× ground-truth length. - For Text2SVG: CLIP text-image similarity and VLM-as-judge replace DreamSim.

GRPO computes group-centered advantages (Ai = Ri − mean(R)), optimizes with PPO clipped ratio, and omits KL regularization (β=0) based on ablation showing it causes early reward saturation without stability benefit.

Key Contributions

  • First application of online RL to inverse rendering code generation (SVG from image/text).
  • Composite rendering-based reward combining pixel-level (L2, L2-Canny), semantic (DreamSim, DreamSim-Canny, CLIP), and code efficiency signals.
  • Demonstrated state-of-the-art Im2SVG performance with RLRF on Qwen2.5-VL-7B, surpassing GPT-4o and Gemini-1.5-Pro.
  • Extension to Text2SVG via caption-only training (no paired SVG supervision) using CLIP+VLM-judge rewards on Qwen3-8B.
  • Ablation evidence that SVG-SFT is prerequisite (RLRF on raw instruction model fails), KL penalty is harmful, and temperature ~1.1 with 64 rollouts is optimal.

Results

  • Qwen2.5VL-7B + RLRF on SVG-Stack-Hard (500 complex SVGs):
  • MSE: 1.03 (vs. 8.60 SFT-only, 16.92 GPT-4o, 17.73 Claude 3.7 Sonnet)
  • SSIM: 95.10 (vs. 79.40 SFT-only, 27.55 GPT-4o)
  • DINO: 98.70 (vs. 93.00 SFT-only, 66.91 GPT-4o)
  • LPIPS: 3.08 (vs. 16.58 SFT-only, 89.00 GPT-4o)
  • Code Efficiency: −334 (near-ground-truth token counts; SFT-only: −2.8k, image processing methods: −7k to −100k)
  • StarVector-1B + RLRF: MSE 3.46 vs. 4.60 base; code efficiency improved from −800 to −127.
  • Qwen2.5VL-3B + RLRF: MSE 4.79 vs. 9.48 SFT-only.
  • OOD generalization: Improved on SVG-Emoji, SVG-Fonts, SVG-Icons despite no training exposure.
  • Text2SVG: Qwen3-8B + RLRF outperforms Qwen3-32B, GPT-4o, and Gemini-1.5-Pro qualitatively (Table 4; quantitative CLIP/LLM-judge scores in appendix).

Limitations

  • Requires SVG-SFT as prerequisite; RLRF alone on instruction-tuned checkpoints yields poor results (MSE 14.23 vs. 4.42 with SFT+RLRF).
  • Training is compute-intensive: RLRF uses 4 nodes × 8 H100 GPUs for ~3 days even for the 500-step, 20k-sample post-training stage.
  • Text2SVG model still struggles with precise Bézier curves and relies on basic geometric primitives.
  • Evaluation limited to fixed 224×224 resolution for StarVector; adaptive resolution used for Qwen2.5-VL but context truncated to 32k (covers 90% of data).
  • DreamSim and CLIP rewards may not fully capture fine-grained geometric or typographic fidelity.
  • Generalization claim for other inverse rendering tasks (HTML, LaTeX/TikZ, CAD) is stated but not empirically validated in the paper.

Relevance to Vision-Language Models

RLRF demonstrates that RL post-training with non-differentiable rendering feedback is a viable and highly effective alternative to SFT-only pipelines for VLMs performing visual code generation—directly addressing the modality gap between token-space supervision and pixel-space correctness. The GRPO formulation with composite rendering rewards is a concrete recipe for extending RL-from-feedback paradigms (à la RLHF/RLAIF) into structured visual-output domains, complementing work on chain-of-thought reasoning RL for VLMs. For researchers tracking VLMs, RLRF establishes that visual feedback can dramatically improve OOD generalization and code efficiency in autoregressive generation, with implications for any task where the model's output can be executed or rendered and compared to a target.