Skip to content

SPRO: Improving Image Generation via Self-Play

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SPRO (Self-Play Reward Optimization) is an annotation-free framework that uses a VLM (Guider-VLM) trained via iterative DPO self-play to generate reward-optimized prompts for diffusion models, then fine-tunes the diffusion model on the resulting high-reward synthetic images. The three-stage pipeline—prompt-space, image-space, and joint optimization—achieves ~30% average improvement across aesthetic, engagement, and user-preference objectives without any human-labeled data.

Problem

Existing methods for aligning diffusion models with human preferences (aesthetics, engagement, user preference) rely on expensive, limited human-annotated datasets or handcrafted prompts. Prompt-only methods explore only human-discoverable strategies; diffusion-only methods (e.g., SPIN Diffusion) are sample-inefficient (>500k images), architecture-specific, and ignore prompt-space exploration. No prior work combines both axes in a unified multimodal self-play paradigm.

Method

SPRO comprises three progressive stages:

  1. SPRO-Prompt: A Guider-VLM (initially pretrained, e.g., LLaVA or Qwen) receives an image–caption pair and a target objective, then generates n candidate reasoning chains and corresponding prompts at varied temperatures. A frozen diffusion model generates images from each prompt; a reward model (PickScore, LAION-Aesthetics, or EngageNet) scores them. Winner–loser pairs (above/below mean reward, CLIP similarity ≥ 0.75) form contrastive data for DPO fine-tuning of the Guider-VLM. This loop repeats 3–4 iterations until reward saturates.

  2. SPRO-Image: The best synthetic images from SPRO-Prompt are paired with their original base captions {p0, I_w} and used to fully fine-tune the diffusion model via MSE reconstruction loss, aligning image-space generation with the target objective.

  3. SPRO-MM: Both the DPO-trained Guider-VLM and the fine-tuned diffusion model operate together—optimized prompts feed into an already preference-aligned generator for joint co-optimization.

Key Contributions

  • First multimodal, multi-model self-play framework coupling a VLM and a diffusion model for joint preference alignment.
  • Annotation-free pipeline achieving state-of-the-art on aesthetics (LAION), user preference (PickScore), and engagement (EngageNet) simultaneously.
  • Reasoning chains in the Guider-VLM provide ~2% average gain over scaling compute alone (Table 5).
  • SPRO-Prompt generalizes zero-shot to unseen diffusion models (SDXL, Flux, DALL·E 3) and unseen reward models (VILA, Improved Aesthetic Scorer, ImageReward).
  • Release of a >1M synthetic image–prompt preference dataset generated entirely via self-play.

Results

  • Aesthetics (Flickr30k, LAION score): SPRO-Prompt 8.60 vs. Promptist 7.23, GPT-4o 7.57, GPT-o3 7.67; 34% gain over base captions (6.40).
  • User preference (Pick-a-Pic, PickScore): SPRO-Prompt 22.42 vs. SPIN Diffusion 22.00, DDPO 21.59, GPT-4o 22.31.
  • Engagement (EngagingImageNet, EOIG score): SPRO-Prompt 83.48/88.36/90.24 (low/mid/high buckets) vs. EOIG-PFT 43.12/56.10/62.30; >2× improvement in the low-engagement bucket.
  • Cross-model transfer (zero-shot): Win rates of 78.9% on SDXL, 80.9% on Flux, 67.67% on DALL·E 3.
  • Human study: SPRO prompts preferred over GPT-4o in 66.7% of cases and over base captions in 78.9%.
  • Sample efficiency: Outperforms SPIN Diffusion using only 30k training examples vs. SPIN's 500k.
  • SPRO-MM further improves over individual stages; combines aesthetics LAION 8.42 and PickScore 22.42 jointly.

Limitations

  • Effectiveness is bounded by the reliability of external reward models, which may not fully capture nuanced human preferences.
  • Multi-objective optimization is handled by independent single-objective runs; dynamic trade-off balancing across simultaneously competing objectives (e.g., maximize aesthetics while maintaining engagement) is not yet supported.
  • The Guider-VLM is trained using SDXL as the frozen generator; cross-model prompt transfer is zero-shot but not explicitly optimized for target architectures.

Relevance to Vision-Language Models

SPRO demonstrates a concrete application of VLMs as autonomous reasoning agents in a generative pipeline—the Guider-VLM does not just caption or classify but actively drives reward-seeking exploration via chain-of-thought prompting and DPO fine-tuning, illustrating how VLMs can serve as intelligent controllers in multi-model systems. The finding that reasoning chains improve generation outcomes (Table 5) is directly relevant to the VLM reasoning literature, connecting to CoT and self-improvement paradigms popularized in text-only LLMs. For VLM alignment researchers, the DPO-based self-play loop provides a scalable recipe for preference alignment without human labels, generalizable beyond image generation to any task where a VLM must produce outputs evaluated by a downstream reward signal. The cross-model generalization of learned prompting strategies also speaks to the question of how well VLM-derived representations and strategies transfer across architectures.