Skip to content

WebVIA: A Web-based Vision-Language Agentic Framework for Interactive and Verifiable UI-to-Code Generation

🕒 Published (v1): 2025-11-09 06:58 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WebVIA is an agentic framework that addresses the gap between static and interactive UI-to-Code generation. It combines a trained exploration agent (WebVIA-Agent) that traverses HTML environments to build an interaction graph, a fine-tuned UI2Code model conditioned on multi-state screenshots, and a task-oriented validation module that executes the generated code to verify functional interactivity.

Problem

Existing VLM-based UI-to-Code methods generate only static HTML/CSS/JavaScript that reproduces visual appearance but lacks interactivity—click handlers, form submissions, state transitions. Generated interfaces cannot be integrated into real-world UI development because they do not respond to user actions.

Method

WebVIA is formalized as a sequential decision-making problem over a structured browser environment (WebEnv, built on Gym API + Playwright). Three components form a pipeline:

  1. Exploration Agent (WebVIA-Agent): At each step \(t\), observes multimodal state \(s_t = (I_t, D_t)\) (screenshot + DOM snapshot), proposes action sequences \(a_t \in \mathcal{A}\), executes them, and verifies whether transitions expose new elements. A hybrid BFS+DFS strategy incrementally builds an interaction graph \(G = (S, T)\). Trained via SFT on GLM-4.1V-9B-base with two datasets: action generation pairs and interaction verification pairs.

  2. UI2Code Model: Conditioned on the full interaction graph (multiple screenshots + validated transitions), generates executable HTML/CSS/JavaScript. Trained on the WebView dataset (11k synthesized webpages) with <think>…</think><answer>…</answer> prompt–response format; Claude is used to synthesize ground-truth code from exploration traces. Two variants: WebVIA-UI2Code-Qwen (Qwen2.5-VL-7B) and WebVIA-UI2Code-GLM (GLM-4.1V-9B).

  3. Validation Module: Executes generated code in WebEnv and replays the task sequences defined by \(G\); passes only if all end-to-end user workflows complete successfully. Uses GPT-5 as the validation model.

Key Contributions

  • First agentic framework for interactive UI-to-Code generation and validation (not just static layout reproduction).
  • Two purpose-trained models: WebVIA-Agent (exploration + verification) and WebVIA-UI2Code (multi-state interactive code generation).
  • Two new benchmarks: UIExplore-Bench (exploration agent evaluation) and UIFlow2Code-Bench (interactive UI2Code evaluation).
  • WebView dataset: 11k synthesized webpages with interaction graphs and executable ground-truth code.
  • Weighted overall exploration metric: \(\text{Overall} = 0.40 \cdot \text{Comp} + 0.35 \cdot \text{Correct} + 0.25 \cdot \text{Dedup}\).

Results

  • Action generation (UIExplore-Bench, 87 samples): WebVIA-Agent F1 = 85.30%, surpassing Claude-Sonnet-4 (83.38%), GPT-5 (81.85%), o4-mini (83.16%); slightly below Gemini-2.5-Pro (81.70% F1 but with overly aggressive element selection).
  • Interaction verification (UIExplore-Bench, 53 samples): WebVIA-Agent overall accuracy = 91.51%, best of all baselines (GPT-5: 90.57%, others ≤88.68%).
  • Pipeline-level exploration: WebVIA-Agent overall score 89.8%, best Completeness (93.1%) and Correctness (97.7%) among all models tested.
  • UIFlow2Code-Bench (interactive UI2Code): WebVIA-UI2Code-Qwen reaches 75.9 and WebVIA-UI2Code-GLM reaches 84.9, approaching Claude-Sonnet-4 performance.
  • Design2Code (static benchmark): +5.2 points for Qwen variant and +4.7 points for GLM variant over respective base models.

Limitations

  • UIFlow2Code-Bench and UIExplore-Bench are newly introduced and not yet externally validated; comparisons rely on self-constructed benchmarks.
  • Training data (WebView) uses synthesized webpages from templates, not real-world sites; generalization to live, noisy web environments is unverified.
  • Validation module depends on GPT-5 as judge, introducing cost and potential inconsistency.
  • Single-step and pipeline evaluations use relatively small sample sizes (87 and 53 samples for agent evaluation).
  • The paper text is truncated; full UI2Code benchmark results and ablation details are not visible.

Relevance to Harnesses / Meta-Harnesses

WebVIA is itself a meta-harness: it orchestrates three specialized sub-components (exploration agent, generation model, validation module) into a closed-loop pipeline where each stage's outputs gate and condition the next. The interaction graph \(G\) plays the role of a structured execution plan that the harness follows and verifies—directly analogous to how meta-harnesses manage task decomposition and verification across sub-agents. The validation module pattern—executing outputs in an environment and checking them against a pre-defined task suite—is a transferable design primitive for any harness that needs to verify LLM-generated artifacts beyond surface-level scoring. The use of Claude to synthesize ground-truth training data from exploration traces is also a notable example of a harness bootstrapping its own training signal.