Skip to content

VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; VisCodex: unified multimodal code generation merging vision and code models

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VisCodex merges a vision-language model (Qwen2.5-VL) with a coding LLM (OpenCodeReasoning-Nemotron) via task-vector arithmetic, bypassing costly pre-training to produce an MLLM with strong multimodal code generation. The authors also release MCD, a 598k-sample instruction-tuning dataset, and InfiBench-V, a 322-question benchmark requiring visual context to answer real-world programming questions. VisCodex-8B surpasses GPT-4o-mini and VisCodex-33B matches GPT-4o on average across four benchmarks.

Problem

Existing MLLMs excel at visual description and reasoning but lack the deep programming knowledge required to generate correct, executable code from visual inputs (UI mockups, charts, screenshots). Training a unified model from scratch is costly, and no large-scale, diverse dataset or realistic benchmark existed to drive and assess multimodal code generation.

Method

Task-vector model merging. Given a shared base model \(\theta_{\text{base}}\) (Qwen2.5 LLM), task vectors are computed as: $\(\tau_{\text{vlm}} = \theta_{\text{vlm}} - \theta_{\text{base}}, \quad \tau_{\text{code}} = \theta_{\text{code}} - \theta_{\text{base}}\)$ The merged language backbone is: $\(\theta_{\text{VisCodex}} = \theta_{\text{base}} + \lambda\,\tau_{\text{vlm}} + (1-\lambda)\,\tau_{\text{code}}\)$ with \(\lambda = 0.7\) selected on MMCode. Merging is applied only to the language backbone; the ViT and cross-modal projection modules are frozen throughout, preserving visual grounding. After merging, only the language backbone is fine-tuned on MCD; the vision encoder and projector remain frozen.

MCD Dataset (598k samples): (1) 200k enhanced HTML webpage image-code pairs regenerated from scratch via GPT-4o seeded by Web2Code screenshots; (2) 210k chart-to-code pairs (164k synthetic from ChartCoder + 46k real GitHub matplotlib scripts filtered via GPT-4o quality scoring); (3) 59k image-augmented StackOverflow QA pairs; (4) 129k algorithmic coding examples from LeetCode, Codeforces, TACO, etc.

InfiBench-V Benchmark (322 questions): Curated from ~1M StackOverflow image-based threads, filtered by GPT-4o to retain questions where the image is indispensable, spanning 13 languages across 5 domains. Evaluated via keyword matching, unit testing, and GPT-4o judging.

Key Contributions

  • VisCodex: task-vector merging to inject code expertise into an MLLM without retraining from scratch, with selective backbone-only merging to preserve visual grounding.
  • MCD: 598k-sample multimodal coding instruction-tuning dataset spanning HTML, chart-to-code, visual QA, and algorithmic code.
  • InfiBench-V: a realistic, human-curated benchmark of 322 visually indispensable programming questions across 5 domains and 13 languages.

Results

  • VisCodex-8B average score 68.8 vs. GPT-4o-mini 65.7; outperforms all open-source models in the 7–15B tier including InternVL3-14B (60.3).
  • VisCodex-33B average score 72.3, on par with GPT-4o (73.3), surpassing Qwen2.5-VL-72B-Instruct (66.9) and InternVL3-78B (65.9) at less than half the parameter count.
  • Design2Code: VisCodex-8B 90.1/90.9 (Low-L/High-L), matching GPT-4o (90.2/90.4); VisCodex-33B 90.5/91.1.
  • ChartMimic (Low-L/High-L): VisCodex-8B 74.8/74.1 vs. baseline Qwen2.5-VL-7B 39.5/38.3; VisCodex-33B 79.3/78.5.
  • MMCode pass@1: VisCodex-8B 11.0 vs. baseline 5.3; VisCodex-33B 15.6 vs. baseline 14.4.
  • Ablation: removing model merging drops Design2Code from 90.1→89.6, ChartMimic Low-L from 74.8→73.4, MMCode from 11.0→6.8 (8B scale).
  • Backbone replacement (1-stage or 2-stage LLaVA-style) consistently underperforms merging on visual benchmarks (e.g., ChartMimic Low-L: 70.4/73.4 vs. 74.8).
  • Among code LLM choices, Nemotron-1.1-7B gives the highest MMCode gain (11.0 vs. 6.8 for general Qwen2.5-7B), while all code-specialized LLMs outperform a general-purpose LLM.

Limitations

  • InfiBench-V contains only 322 questions, limiting statistical power across the 13 languages and 5 domains.
  • Optimal \(\lambda\) is selected on a single benchmark (MMCode) and may not generalize across domains.
  • MCD relies heavily on GPT-4o for data generation and quality filtering, inheriting any biases or errors from that model.
  • The merging formula requires that the VLM's language backbone and the coding LLM share the same architecture and base model (Qwen2.5), restricting applicability to architecturally compatible pairs.
  • Performance gains on MMCode pass@1 remain modest even after merging (11.0), suggesting algorithmic visual reasoning is not fully solved.

Relevance to Vision-Language Models

VisCodex directly addresses the gap between VLM visual grounding and code generation by showing that task-vector merging—a lightweight, data-free operation—can transfer specialized capabilities into an MLLM without disrupting its visual alignment. For VLM researchers, this establishes model merging as a practical alternative to full multimodal pre-training or architecture replacement, with explicit evidence that backbone replacement degrades visual grounding while merging preserves it. The release of MCD and InfiBench-V provides new training and evaluation infrastructure for the emerging subfield of multimodal code generation, which extends VLM capabilities into structured, executable output spaces (HTML, chart code, programming QA).