Skip to content

The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric

🕒 Published (v1): 2026-07-20 17:59 UTC · Source: Arxiv · link

Why this paper was selected

Jun-Yan Zhu + Shechtman + Hertzmann; text-prompted multi-sense perceptual metric, novel

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Existing perceptual similarity metrics (LPIPS, DreamSim) collapse visual similarity into a single scalar, ignoring context-dependence. This paper introduces TPIPS, a text-prompted image perceptual similarity metric trained on 1M human similarity judgments across 25K triplets annotated under free-form aspect conditions. TPIPS substantially narrows the gap between model and human agreement on aspect-conditioned similarity tasks and generalizes to out-of-distribution image domains.

Problem

No existing perceptual metric can condition similarity judgments on a specified visual aspect (e.g., color, pose, background). Prior conditional similarity methods (GeneCIS, FocalLens, Omni-Attribute) rely on VLM-generated or caption-mined supervision rather than direct human perceptual judgments, leaving a measurable alignment gap with human consensus. No large-scale human-annotated dataset existed for aspect-conditioned perceptual similarity evaluation.

Method

The authors collect a synthetic odd-one-out (OOO) dataset of 24K image triplets with 1M+ human votes, where each triplet is annotated under multiple free-form aspect conditions (e.g., "lighting warmth", "cow pose"). Triplets are generated by using an LLM to vary T2I prompts (FLUX-Reason-6M), with a VLM refining candidate aspect lists; annotators prune aspects via a "can't tell" filter. The model \(f_\theta(x_1, x_2 \mid c)\) is trained with a softmax choice model over triplet pairs under cross-entropy loss (Equation 2). Three fusion architectures are explored:

  • Late fusion: images encoded independently with aspect-conditioned prompt; cosine similarity of final embeddings.
  • Mid fusion: per-layer internal VLM activations compared via negative weighted \(\ell_2\) distance with text-conditioned per-channel weights \(w_\ell(c)\) from an MLP; backbone fine-tuned with LoRA.
  • Early fusion: both images concatenated into a single sequence \([\,c \;\|\; x_1 \;\|\; x_2 \;\|\; r_1 \;\|\; r_2\,]\) allowing full cross-image attention; similarity from cosine of two learnable register token readouts, with attention masking enforcing symmetry and identity by construction.

Generalization is assessed on a separate 2AFC out-of-distribution set from four vision/graphics tasks (image editing, compositing, novel view synthesis, image-to-3D).

Key Contributions

  • A large-scale dataset of ~1M human similarity judgments over 25K triplets annotated under 257K triplet–aspect combinations, released publicly.
  • A benchmark revealing a substantial gap between frontier VLMs/embedding models and human consensus on aspect-conditioned perceptual similarity.
  • TPIPS: a fine-tuned VLM perceptual metric in three fusion variants (late, mid, early) that conditions similarity on free-form text aspects.
  • Demonstrated downstream applications: aspect-conditioned nearest-neighbor retrieval, compositional multi-query retrieval, and fine-grained evaluation of generative vision models.
  • An out-of-distribution 2AFC evaluation protocol spanning four real vision/graphics pipelines.

Results

  • Human consensus ceiling: 67.5% (OOO, chance = 33.3%); 89.3% (2AFC OOO).
  • Best baseline on OOO: Gemini-3.1-Pro at 58.4%; best open-weight: Qwen3-VL-Embed at 54.5%.
  • TPIPS (early fusion) on OOO: 64.7%, reducing the gap to human consensus from 9.1% to 2.8%.
  • TPIPS (early fusion) on out-of-distribution 2AFC: 79.3%, reducing error from 15.8% to 10.0% vs. best baseline.
  • Classic aspect-agnostic metrics: LPIPS 46.2%, DreamSim 50.5% on OOO.
  • All three TPIPS fusion variants outperform all baselines on both evaluation sets; early fusion is marginally best.

Limitations

  • Training data is entirely synthetic (T2I-generated images), which may not fully capture the diversity of natural image distributions.
  • The larger residual gap on the out-of-distribution 2AFC set (10.0% below ceiling vs. 2.8% in-domain) indicates imperfect generalization to real algorithm outputs.
  • Mid-fusion performs poorly for off-the-shelf baselines because existing VLMs are not designed to produce informative mid-level features; controlled comparisons partially mitigate this.
  • The paper is truncated before full ablation and architecture comparison results are reported.
  • "Can't tell" filtering may introduce selection bias toward aspects that are more perceptually salient or linguistically describable.

Relevance to Vision-Language Models

This paper directly stress-tests frontier VLMs (GPT-5.4, Gemini-3.1-Pro, Qwen3-VL, InternVL3.5) on a structured perceptual reasoning task, revealing that even the strongest proprietary systems fall substantially below human consensus when similarity must be conditioned on a specified visual aspect. The work demonstrates a concrete fine-tuning pathway — repurposing VLM backbones as aspect-conditioned perceptual metrics via late/mid/early fusion — that significantly closes this gap. For researchers studying VLM evaluation, grounding, or instruction-following, TPIPS provides both a diagnostic benchmark and a reusable metric that quantifies how well VLMs translate text-specified semantic dimensions into image-level similarity judgments.