Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification

🕒 Published (v1): 2026-03-27 17:50 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Vision2Web is a hierarchical benchmark (193 tasks, 918 prototype images, 1,255 test cases) for evaluating multimodal coding agents on visual website development across three progressive levels: static webpage generation, interactive multi-page frontends, and long-horizon full-stack systems. It introduces a workflow-based agent verification paradigm that orchestrates a GUI agent verifier and a VLM judge through structured dependency graphs to assess both functional correctness and visual fidelity. State-of-the-art models (Claude-Opus-4.5, GPT-5, Gemini-3-Pro-Preview) show severe performance degradation on full-stack tasks, revealing fundamental capability gaps.

Problem

Existing coding-agent benchmarks have three compounding deficiencies: (1) task formulation is narrowly issue-driven (SWE-Bench family), evaluating incremental edits rather than holistic end-to-end development; (2) multimodal benchmarks (Design2Code) are restricted to single-page static reproduction, missing interactive and full-stack complexity; (3) automated verification is unreliable — loosely specified objectives cause GUI agents to explore unconstrained execution paths, breaking reproducibility for long-horizon systems.

Method

Dataset construction. Websites are sourced exclusively from the C4 validation set to prevent contamination, then filtered through three stages: DOM-level structural assessment (retaining 63,515 pages), VLM-based content/quality scoring (reducing to 7,391), and expert manual review for balanced category coverage.

Hierarchical task formulation. Three levels of increasing complexity: - Level 1 (Static Webpage): reproduce a single page at desktop/tablet/mobile resolutions from prototype images. - Level 2 (Interactive Frontend): generate multi-page frontends with correct navigation flows from multiple prototypes + inter-page relationship text. - Level 3 (Full-Stack Website): interpret structured requirement documents + prototypes, manage application state, perform integrated debugging, deliver complete systems.

Workflow-based agent verification. End-to-end testing is formalized as a directed dependency graph where nodes are verification sub-procedures and edges encode sequential state dependencies. The graph is instantiated into test workflows — agent-executable subgraphs — by two principles: (a) decoupling nodes spanning separate functional modules to prevent error propagation; (b) grouping nodes sharing application context to reduce redundant setup.

Verification nodes come in two types: - Functional Verification Node \(n_i = \langle O_i, A_i, V_i \rangle\) (objective, guided actions, validation criteria): executed by a GUI agent (WebVoyager protocol, instantiated with GLM-4.6V) with full history context \(C_i = \{H_{<i}, O_i, A_i, V_i\}\); reports Functional Score (FS). - Visual Verification Node \(n_i = \langle P_i \rangle\) (target prototype): evaluated by a VLM judge (Gemini-3-Pro-Preview) via component-level comparison against predefined visual rubrics; reports Visual Score (VS).

Agent-assisted annotation. Test cases for Levels 1–2 are semi-automatically generated by Claude Code from prototypes. Level 3 uses an expert-in-the-loop strategy: PhD researchers draft high-level workflow skeletons encoding task dependencies and execution order; Claude Code then translates these into executable interaction sequences and verifiable criteria.

Key Contributions

• Hierarchical task design spanning static, interactive, and full-stack website development with explicit level-wise capability disentanglement.
• Workflow-based agent verification paradigm: a dependency-graph abstraction that constrains GUI-agent execution via structured workflows while preserving interaction flexibility; implementation-agnostic by design.
• Dual verifier architecture: complementary GUI agent verifier (functional correctness, FS) and VLM judge (visual fidelity, VS) unified under a single evaluation framework.
• Agent-assisted annotation pipeline combining expert drafting with Claude Code refinement, enabling scalable test-case generation for long-horizon tasks.
• Empirical analysis of 8 state-of-the-art models (Claude-Opus-4.5, GPT-5, Gemini-3-Pro-Preview, Gemini-3-Flash-Preview, Claude-Sonnet-4.5, Seed-1.8-VL, Qwen3-VL-32B/8B) across two coding-agent frameworks (OpenHands, Claude Code).

Results

• Performance degrades across levels for all agents. Under OpenHands, Gemini-3-Pro-Preview achieves desktop VS = 63.3 on static pages but drops to VS = 11.7, FS = 22.6 on full-stack tasks (DSR 77.8%).
• Claude-Opus-4.5 is the strongest overall under both frameworks: OpenHands full-stack VS/FS = 38.4/57.6; Claude Code full-stack VS/FS = 34.3/53.1.
• Mobile and tablet layouts lag desktop by 10–20% VS even for top agents; larger/denser prototype images further degrade visual scores.
• Seed-1.8-VL and Qwen3-VL models fail entirely on full-stack tasks (VS = 0, FS = 0 under OpenHands).
• Framework matters: Most non-Claude models score higher under OpenHands than Claude Code; Claude models perform comparably or better under Claude Code.
• Category variation: SaaS platform tasks are the hardest (Claude-Opus-4.5: VS/FS = 22.9/39.9); Public Service tasks are easiest (56.9/60.0).
• Functional category breakdown (OpenHands, Claude-Opus-4.5): Navigation & Routing 66.3, Authentication 61.8, State Management 43.2, File & Media Operations 33.3.

Limitations

• Small full-stack subset: only 27 tasks at Level 3; conclusions about long-horizon capability are statistically limited.
• Evaluator bias: both the GUI verifier (GLM-4.6V) and VLM judge (Gemini-3-Pro-Preview) are themselves imperfect models; errors in verification can mask or inflate true agent performance.
• Containerized environment: may not fully capture real-world deployment variance (network, OS, third-party APIs).
• Three-iteration cap on deployment attempts limits analysis of agents that converge slowly.
• Failure-mode analysis is truncated in the provided text; the depth of error categorization is unclear.
• No human-performance ceiling reported; the gap between agents and human developers is unmeasured.

Relevance to Harnesses / Meta-Harnesses

Vision2Web's workflow-based agent verification paradigm is itself a meta-harness: it orchestrates two specialized sub-agents (a GUI executor and a VLM judge) through a structured dependency graph, controlling execution order, state sharing, and verification logic — the defining characteristics of a harness coordinating other agents. The directed dependency graph abstraction for test workflows directly parallels harness design patterns (task decomposition, sequential state threading, pluggable verifier modules). The agent-assisted annotation pipeline is a harness-of-harnesses instance: Claude Code acts as an annotation agent driven by expert-authored skeletons, producing the test scaffolding that the verification harness later executes. For researchers building or evaluating harnesses, Vision2Web provides both a concrete implementation blueprint for multi-agent evaluation orchestration and a benchmark to measure how well harness-driven coding agents handle end-to-end software construction.