Skip to content

Measuring And Improving Engagement of Text-to-Image Generation Models

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Existing text-to-image evaluation metrics (aesthetics, CLIPScore, PickScore, ImageReward) correlate near-randomly with real-world viewer engagement (Pearson r ≈ 0.02–0.08). The authors introduce EngagingImageNet (168M enterprise tweets with like counts), train EngageNet—a fine-tuned LLaVA-1.5 VLM—to predict engagement (r = 0.62), and show three methods for optimizing diffusion models toward higher engagement.

Problem

No large-scale dataset or reliable metric captures viewer engagement for generated images. Metrics optimized for creator preferences (PickScore, HPS) or aesthetics do not predict downstream marketing utility (clicks, likes, purchases), leaving a gap between image quality and commercial outcome alignment.

Method

Dataset: EngagingImageNet collects 168M tweets from 10,135 verified enterprise Twitter accounts (2007–2023) via the Twitter API; images are binned into low/medium/high engagement by per-account normalized likes percentile, yielding 837K labeled samples with LLaVA-extracted captions, colors, tones, and metadata.

EngageNet: Visual instruction fine-tuning of LLaVA-1.5 13B on EngagingImageNet. The instruction includes company handle, posting date, image description, colors/tones, and resolution. A two-layer MLP regression head on the decoder's final hidden states adds an auxiliary MSE loss (λ=0.1) alongside cross-entropy, improving sensitivity to score magnitude.

Engagement optimization (three methods): 1. Prompt conditioning (run-time): FAISS retrieval of semantically similar captions from high-engagement images; retrieved caption replaces the original if cosine similarity > threshold τ. 2. Preferred finetuning (PFT, train-time): Fine-tune Stable Diffusion 1.4 U-Net for 50 epochs on high-liked images using the standard denoising score-matching loss. 3. RLHF (train-time): DDPO policy gradient with two reward variants—EngageNet directly scoring generated images (RLHF-ES), or EngageNet predicting verbalized design specifications that the generated image should match (RLHF-DSG).

Engagement Arena: Elo-rated tournament among text-to-image models using EngageNet-Oracle as the judge, modeled on LMSYS Chatbot Arena.

Key Contributions

  • EngagingImageNet: first large-scale dataset (28.5M media-containing tweets, 837K filtered samples) with ground-truth viewer engagement for image generation research.
  • EngageNet: engagement-aware VLM (LLaVA-1.5 fine-tuned with MSE auxiliary loss) achieving Pearson r = 0.62 vs. actual likes; cross-platform transfer to Flickr achieves r = 0.53 after minimal fine-tuning.
  • Empirical demonstration that all standard T2I evaluation metrics fail to capture engagement (r ≤ 0.08).
  • Three complementary engagement optimization strategies, characterized with ablations.
  • Engagement Arena: first automated Elo benchmark for ranking T2I models on engagement.

Results

  • EngageNet (full model): Pearson r = 0.62 with account-normalized likes; Oracle variant (trained on all data) achieves r = 0.87.
  • GPT-4V (5-shot ICL): r = 0.13—best baseline; all standard metrics ≤ 0.08.
  • Prompt conditioning: +12.4% average engagement across all models tested (range 6–15%); SD 3 Medium with improved prompts achieves oracle score 55.08 vs. 50.66 baseline.
  • Preferred finetuning (PFT) on SD 1.4: +10.49% / +8.54% / +11.23% engagement on high/medium/low buckets vs. base SD 1.4.
  • RLHF-ES: +3.78% / +2.72% / +3.70% (smaller gains than PFT).
  • RLHF-DSG: +3.15% / +2.48% / +1.88% (smallest gains).
  • Engagement Arena: SD 3 Medium ranks highest (46% win rate vs. ground truth); SDXL-DPO and Flux.1-dev follow; DALL·E-2 (6.5B) underperforms PixArt-family (600M) and EOIG-SD-PFT (860M).
  • FID, CLIP score, and PickScore remain nearly unchanged across engagement-optimized vs. baseline models, confirming metric non-alignment.

Limitations

  • EngageNet is trained on enterprise Twitter data; generalization to other domains (consumer content, product types) is partially validated on Flickr but not comprehensively.
  • Engagement is measured only by likes; richer signals (clicks, purchases, dwell time) are unavailable via Twitter API.
  • RLHF experiments are limited to SD 1.4; inherent quality ceiling of this backbone constrains improvement headroom, as evidenced by the gap to larger arena models.
  • Prompt conditioning requires a corpus of high-engagement captions as a retrieval index, limiting cold-start applicability.
  • EngageNet's contextual inputs (company handle, date) are marketing-specific; applicability in general-purpose image generation is unclear.
  • Persuasion-strategy biases (social appeal, social identity) emerge as side effects of engagement training, raising ethical considerations not fully addressed.

Relevance to Vision-Language Models

EngageNet directly instantiates a VLM (LLaVA-1.5) as a reward model for a downstream generative pipeline, demonstrating that instruction-finetuned VLMs can be adapted to predict nuanced behavioral outcomes—viewer engagement—beyond standard vision-language alignment tasks. The work is relevant to the growing body of VLM-as-judge research, showing that contextual metadata (brand, time) is critical for such predictions and that standard VLM prompting (GPT-4V) falls far short without task-specific fine-tuning. For researchers tracking VLMs, this paper establishes both a dataset and a training recipe for engagement-aware multimodal scoring, and its cross-platform transfer results contribute evidence about VLM generalization to non-obvious behavioral proxies.