Skip to content

Through the PRISM: Principle-Aware, Interpretable, and Multi-Scale Evaluation of Visual Designs

🕒 Published (v1): 2026-05-30 07:44 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; PRISM: principle-aware multi-scale evaluation of visual designs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PRISM is a 110K-sample benchmark derived from the Crello corpus that systematically perturbs professional layouts along five disentangled design principles (coherence, readability, contrast, alignment, overlap) to enable principle-specific, interpretable evaluation of visual design quality. The authors show that current MLLMs (Qwen-2.5-VL, GPT-4o-mini) are largely insensitive to these targeted degradations, while GPT-4o detects global changes but lacks fine-grained principle separation. They propose a multi-scale framework pairing SigLIP-v2-based binary scorers with an instruction-tuned localiser and a beam-search editing pipeline for iterative, interpretable layout refinement.

Problem

Existing design evaluation methods collapse multi-principle reasoning into a single scalar score (aesthetic preference, layout similarity), offering no diagnostic signal about which principle is violated or where in the layout. MLLM-based free-form critiques are stochastic, non-reproducible, and lack grounding in explicit design principles—they behave as subjective critics rather than consistent evaluators. No controlled benchmark existed to measure model sensitivity to isolated principle violations.

Method

Dataset construction (PRISM). Starting from ~20K training / 2K validation professional designs in the Crello corpus (with structured metadata: geometry, text attributes, color palette), each design \(x\) is perturbed independently per principle \(p\) to produce \(x'_p = x + \Delta_p\). GPT-4o handles semantic edits (text replacement, icon/background swaps, RGB color suggestions) via metadata-only prompts (no full poster image), while geometric/typographic attributes (coordinates, font size) are scaled directly via Crello metadata. This yields 100K training and 10K validation paired samples.

PRISM-Scorer. A SigLIP-v2 vision encoder is first design-aware fine-tuned on 420K layout–caption pairs (Creatidesign + Crello) with the text encoder frozen. Five independent binary classifiers are then trained on top of frozen embeddings \(\phi(x)\): $\(g_p(x) = \sigma(\phi(x)^\top W_p)\)$ Each classifier is trained with binary cross-entropy on balanced composed/perturbed sets (40K train, 4K test).

PRISM-Localiser. For perceptual principles (readability, contrast, overlap), Qwen-2.5-VL (7B) is instruction-tuned on multiple-choice prompts: given the layout image and a candidate set \(\mathcal{O} = \{o_1, \ldots, o_n\}\) of element IDs with content, the model predicts the violating subset \(\hat{\mathcal{O}}_p \subseteq \mathcal{O}\), trained with cross-entropy against ground-truth annotations. Output is constrained to element IDs (no free-form text). For global principles (coherence, alignment), prompt-based evaluation with MLLMs is used (theme inference → per-component structured queries).

Editing pipeline. A beam-search editor (beam size = 2) proposes minimal candidate edits, scores them with PRISM-Scorer, and retains top-scoring refinements.

Key Contributions

  • PRISM benchmark: 100K/10K train/val paired layouts isolating principle-specific violations across five design dimensions, with ground-truth localisation annotations for perceptual principles.
  • PRISM-Scorer: principle-disentangled SigLIP-v2-based binary classifiers providing quantitative, per-principle quality scores.
  • PRISM-Localiser: instruction-tuned Qwen-2.5-VL that outputs element-level violation identifiers for spatially grounded, reproducible feedback.
  • Empirical analysis showing SOTA MLLMs fail to discriminate targeted principle degradations, with PRISM-Scorer outperforming GPT-4o in fine-grained principle disentanglement.
  • Beam-search editing pipeline demonstrating score improvement from 0.38 → 0.58 on a coherence example.

Results

PRISM-Scorer (binary classification, held-out val): - Coherence: Precision 0.72, Recall 0.73, F1 0.73, AUC 0.80 - Readability: Precision 0.73, Recall 0.73, F1 0.73, AUC 0.81 - Contrast: Precision 0.76, Recall 0.75, F1 0.75, AUC 0.83 - Alignment: Precision 0.76, Recall 0.76, F1 0.76, AUC 0.83 - Overlap: Precision 0.72, Recall 0.72, F1 0.72, AUC 0.80

Model sensitivity (\(\Delta\)-score, original vs. perturbed): - PRISM-Scorer outperforms GPT-4o across all five principles in discriminating perturbed from composed designs (e.g., coherence: PRISM-Scorer \(\Delta\)=0.32 vs. GPT-4o \(\Delta\)=0.25; alignment: 0.31 vs. 0.24). - Qwen-2.5-VL and GPT-4o-mini show near-zero sensitivity to targeted perturbations. - GPT-4o shows broad sensitivity but fails to separate fine-grained principles.

Human study (disentanglement): 15 participants rating 10 poster pairs per principle on a 1–7 scale confirm strong diagonal responses—perturbations primarily degrade their target principle while leaving others intact.

Limitations

  • Ground-truth localisation annotations are only available for perceptual principles (readability, contrast, overlap); coherence and alignment lack definitive spatial ground truth.
  • PRISM is derived solely from Crello, a single design corpus; generalization to other layout domains (e.g., UI design, scientific figures) is unverified.
  • Complete principle independence during perturbation is acknowledged as inherently challenging; collateral effects on non-target principles are minimized but not eliminated.
  • The localiser is evaluated only qualitatively for global principles (coherence, alignment) due to absence of ground-truth annotations.
  • Beam-search editing relies on GPT-4o as the prompting-based localiser for coherence, introducing API dependency and cost.
  • Human study is small (15 participants, 10 pairs per principle).

Relevance to Vision-Language Models

PRISM directly benchmarks MLLM sensitivity to controlled visual perturbations in a structured domain, revealing that leading VLMs (Qwen-2.5-VL, GPT-4o-mini, GPT-4o) lack the fine-grained perceptual disentanglement needed for design evaluation—a finding that generalizes to any task requiring principle-specific visual reasoning rather than holistic scene understanding. The instruction-tuning of Qwen-2.5-VL for discrete element-ID selection (rather than free-form critique) demonstrates a reproducibility-oriented prompting paradigm applicable to other VLM grounding tasks. This work connects to broader efforts on spatially grounded VLM evaluation and controlled benchmark construction for probing specific visual capabilities, complementing benchmarks like MMBench and POPE that test coarser perceptual competencies.