Skip to content

UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation

🕒 Published (v1): 2026-07-07 14:08 UTC · Source: Arxiv · link

Why this paper was selected

UI-to-app generation benchmark; visual interaction inference; novel multimodal code-gen eval

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

UI2App is the first benchmark that evaluates interaction inference — the ability of VLMs to recover application behavior from static screenshots alone, without any textual or behavioral specification. Evaluated across six frontier VLMs on 45 multi-route web applications, even the best model (Claude Sonnet 4.6) achieves only 39.3/100 on the Interaction Inference Score (IIS). Visual fidelity and interaction-inference capability are empirically decoupled: the VFS leader (Gemini 3.1 Pro Preview, 78.1) scores only 7.5 on IIS, placing fourth.

Problem

Existing screenshot-to-code benchmarks measure only visual fidelity (pixel/DOM similarity to a reference), ignoring whether generated artifacts function correctly as interactive applications. No prior benchmark evaluates interaction inference under image-only input — i.e., recovering interaction logic (state management, CRUD, cross-route persistence) from static screenshots when no action trace, caption, or behavioral description is provided.

Method

Dataset. UI2App curates 45 multi-route web applications (327 screenshots total, 4–14 screenshots per app, mean 7.3) from open-source GitHub repositories. A three-level expert rubric selects apps for page-level discriminability, application-level complexity, and corpus-level diversity across four categories (Content, Admin, Transaction, Specialty). Screenshots are captured headlessly at 1440×900 post-hydration, deduplicated by perceptual hashing, and organized into state-coherent sets representing distinct routes.

Generation. Models are given only screenshots and must emit runnable React + TypeScript source code via a two-pass protocol: (1) emit a file plan, (2) generate per-file content. Up to three self-debug rounds feed build stderr back to the model.

Evaluation pipeline. Four metrics: - EXEC@1 / EXEC@3: binary pass rate for production build + home-route render, with up to three error-feedback repair rounds. - NRS (Navigation Reachability Score): fraction of input-screenshot routes reachable from the home page via visible navigation, verified by human annotation because modal/nested routing cannot be reliably enumerated programmatically. - VFS (Visual Fidelity Score): judge-free DOM block-level matching via optimal bipartite pairing, averaging four sub-metrics (Size, Text, Position, Color). - IIS (Interaction Inference Score): the core metric.

IIS taxonomy and scoring. Interactions are organized into seven categories (C01 toggle, C02 expand/collapse, C03 list operations, C04 data CRUD, C05 form validation, C06 notification, C07 cross-route state) and evaluated along three axes: coverage \(y_i \in \{0,1\}\), result \(r_i \in \{1, 0.5, 0\}\) (working/partial/failed), and scope \(s_i^{\text{gen}} \in \{S1\text{ UI-state}, S2\text{ data-state}, S3\text{ cross-route persistence}\}\) with linear weights \(w_{S1}=1, w_{S2}=2, w_{S3}=3\). The per-application quality score and model-level IIS are:

\[QS_{a,m} = \frac{\sum_{i \in G_a} y_i \cdot r_i \cdot \min\!\left(w_{s_i^{\text{gen}}},\, w_{s_i^{\text{ref}}}\right)}{\sum_{i \in G_a} w_{s_i^{\text{ref}}}}, \quad IIS_m = \frac{100}{N}\sum_{a=1}^N QS_{a,m}\]

where \(G_a\) is the reference set of interactions inferable from screenshots, making IIS recall-oriented. Three annotators score generated artifacts with inter-annotator agreement Krippendorff \(\alpha \in [0.72, 0.84]\).

Key Contributions

  • UI2App benchmark: 327 screenshots in 45 state-coherent sets targeting interaction inference from image-only input — the first benchmark of this type.
  • IIS metric and interaction taxonomy: a rubric-based, implementation-agnostic metric across seven interaction categories and three scope levels, admitting any valid realization without requiring a single reference execution trace.
  • Four-metric evaluation protocol: EXEC@1/3, NRS, VFS, and IIS in a fully automated end-to-end pipeline with human annotation for behavioral judgments.
  • Empirical finding: visual fidelity and interaction-inference capability are decoupled — the VFS leader (Gemini) scores 5.2× lower on IIS than the IIS leader (Claude Sonnet 4.6).
  • Scaling study: Qwen2.5-VL ladder (3B → 72B) establishes scaling baselines for image-only interaction inference.

Results

  • Overall IIS: Claude Sonnet 4.6 leads at 39.3; Kimi K2.5 second at 20.7; Gemini 3.1 Pro Preview fourth at 7.5 (5.2× behind leader).
  • EXEC@3: Claude Sonnet 4.6 and Gemini 3.1 Pro Preview both reach 100%; GLM-4.6V reaches only 35.6%.
  • VFS: Gemini leads at 78.1; Claude second at 75.7; GLM-4.6V lowest at 22.6.
  • NRS: Claude leads at 87.3; GLM-4.6V lowest at 22.3.
  • Cross-route state (S3): Three of six models score exactly 0.0; the best (Claude) reaches only 21.6/100.
  • S-tier IIS spread: S1 spread across models = 42.3 points; S2 = 30.3; S3 = 21.6, showing capability collapses at higher scope.
  • Self-debug heterogeneity: GPT-5.4 recovers ~+18 EXEC points via self-debug but still ranks 5th on IIS (6.7), confirming self-debug helps build success, not interaction inference.
  • VFS conditioning: conditioning on EXEC@3-passing apps shrinks the VFS range to under 20 points; GLM-4.6V's conditional VFS rises from 22.6 to 59.8, showing headline VFS spread is largely build-driven.
  • Gemini zero scores: scores exactly 0.0 on C04 (CRUD), C06 (notification), and C07 (cross-route state) despite leading VFS.

Limitations

  • 45 applications is a small corpus, and curation prioritizes diversity over scale; memorization effects for popular open-source UI templates cannot be fully ruled out.
  • Human annotation for NRS and IIS does not scale and introduces potential annotator bias despite arbitration (Krippendorff \(\alpha\) floor 0.72 is modest for a three-point scale).
  • The fixed React + TypeScript scaffold may favor models pre-trained heavily on that stack, limiting generalizability to other frontend frameworks or mobile UIs.
  • IIS scope weights (\(w_{S1}=1, w_{S2}=2, w_{S3}=3\)) are linearly assigned without empirical justification for the relative difficulty of each scope level.
  • Only six frontier VLMs evaluated; the Qwen2.5-VL scaling ladder uses a different family than most closed frontier models, limiting cross-family scaling conclusions.
  • The paper text is truncated; findings 3 and 4 of the interaction deep-dive are not fully included.

Relevance to Vision-Language Models

UI2App directly stress-tests VLMs on a capability gap that visual-fidelity benchmarks conceal: inferring latent application behavior (state machines, cross-route data flow) from static visual signals alone, which requires spatial-semantic reasoning beyond pixel reconstruction. The empirical decoupling of VFS and IIS rankings provides a concrete, reproducible signal that current VLM architectures do not uniformly couple visual understanding with behavioral reasoning — a finding directly relevant to research on grounded visual reasoning and code generation with VLMs. The benchmark also reveals a frontier-wide bottleneck at S3 cross-route state, pointing toward persistent gaps in VLMs' ability to reason about multi-page application state from image evidence. For VLM researchers, UI2App establishes baseline numbers and a fine-grained taxonomy that can guide targeted training, prompting, or architectural improvements aimed at bridging the visual-to-behavioral inference gap.