Skip to content

FLUX-Reason-6M & PRISM-Bench: A Million-Scale Text-to-Image Reasoning Dataset and Comprehensive Benchmark

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; million-scale T2I reasoning dataset and benchmark from top labs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FLUX-Reason-6M is a 6-million-image, 20-million-caption bilingual dataset paired with Generation Chain-of-Thought (GCoT) annotations structured across six T2I reasoning dimensions. PRISM-Bench is a 700-prompt, seven-track evaluation benchmark that uses VLMs (GPT-4.1, Qwen2.5-VL-72B) as judges for human-aligned scoring of prompt–image alignment and aesthetics. Together they target the open-source T2I gap relative to closed-source systems by providing structured reasoning supervision and discriminative evaluation.

Problem

Existing open-source T2I datasets are flat image–caption pairs (e.g., LAION) lacking structured compositional reasoning signals, producing a training deficit for complex scene synthesis. Existing benchmarks cover only a few dimensions, rely on saturatable metrics like CLIP scores or object detectors, and omit dimensions such as creative imagination and emotional expression—leaving no reliable way to differentiate top models or guide improvement.

Method

Dataset (FLUX-Reason-6M): A four-stage pipeline running on 128 A100s over four months: 1. Synthesis: Rewrite LAION-Aesthetics captions with a VLM, then generate images with FLUX.1-dev. Supplement with targeted augmentation—progressive imagination cultivation via Gemini-2.5-Pro seed prompts expanded by Qwen3-32B at high temperature, and a mining–generation–synthesis pipeline (Qwen2.5-VL-32B on LAION-2B) for text-rendering images. 2. Quality filtering: Qwen-VL removes blur/artifacts and scores each image 1–10 on all six characteristics (Imagination, Entity, Text rendering, Style, Affection, Composition); images failing per-characteristic thresholds are dropped (8M → 6M). 3. Dense captioning & GCoT: Per-characteristic captions generated by Qwen-VL; a VLM then synthesises a GCoT that integrates all category captions into a structured narrative covering layout, typography, spatial relations, emotional tone, and style rationale. 4. Caption integration & bilingual translation: LAION-Aesthetics captions passing a VLM alignment check are merged; the full corpus (~20M captions) is translated into Chinese, preserving English strings for text-rendering tasks.

Benchmark (PRISM-Bench): 700 prompts (100 per track) across seven tracks. Per track: 50 prompts sampled via k-means (k=50) clustering of the top-10k dataset prompts, 50 curated hard-case prompts. Human review by five domain experts ensures grammatical, logical, and T2I-feasibility quality. Evaluation uses GPT-4.1 and Qwen2.5-VL-72B as judges, each returning a 1–10 alignment score (track-specific rubric) and a 1–10 uniform aesthetic score. Final model score = mean over seven tracks of the mean of alignment and aesthetic averages.

Key Contributions

  • FLUX-Reason-6M: 6M FLUX.1-dev images with 20M bilingual captions; first multi-dimensional, million-scale GCoT annotation corpus for T2I.
  • GCoT formulation: Explicit generation chain-of-thought that decomposes scene intent across six characteristics, providing richer training signal than layout-only chain-of-thought (e.g., GoT).
  • PRISM-Bench: Seven-track, 700-prompt bilingual benchmark (including a Long Text track using GCoT prompts) with dual-axis VLM judging (alignment + aesthetics).
  • Large-scale evaluation: 19 models benchmarked, including Gemini2.5-Flash-Image, GPT-Image-1, Qwen-Image, FLUX series, SD series, and HiDream, revealing cross-model and cross-track performance gaps.
  • Public release of dataset, benchmark, and evaluation code.

Results

  • GPT-Image-1 [High]: overall 86.3 (best); best on Entity (88.2), Text rendering (74.5), Style (93.1), Composition (92.8).
  • Gemini2.5-Flash-Image: overall 85.3 (second); best on Imagination (88.6) and Affection (92.1).
  • Best open-source models: Qwen-Image 79.9, SEEDream 3.0 79.6—both ~6 points behind top closed-source.
  • Hardest tracks: Text rendering (SD1.5: 20.6, GPT-Image-1: 74.5) and Long Text are lowest across all models.
  • Easiest tracks: Style and Composition are relatively saturated even for older models (SD1.5 Style 55.3 vs. GPT-Image-1 93.1).
  • SDXL → SD3.5-Large progression: 60.4 → 73.7, showing steady open-source improvement over generations.
  • Closed-source vs. open-source gap persists: even the best open-source (Qwen-Image 79.9) trails GPT-Image-1 by ~6.4 points overall.

Limitations

  • All training images are FLUX.1-dev synthetics; the dataset does not cover the distribution of real-world photography or other generation architectures, potentially biasing models trained on it toward FLUX aesthetics.
  • GCoT construction relies on VLM outputs (Qwen-VL, Gemini-2.5-Pro), which may inherit VLM hallucination or stylistic biases.
  • PRISM-Bench uses 100 prompts per track; statistical power for fine-grained within-track comparisons may be limited.
  • VLM judges (GPT-4.1, Qwen2.5-VL-72B) are closed-source or large-scale; scores may shift as judge models are updated, reducing longitudinal comparability.
  • Chinese benchmark (PRISM-Bench-ZH) was evaluated on only five models; cross-lingual coverage is incomplete.
  • Dataset creation cost (15,000 A100-days) is prohibitive for replication or domain extension.

Relevance to Vision-Language Models

VLMs play a triple role in this work: as data curators (quality filtering, multi-dimensional scoring, dense captioning, GCoT synthesis), as evaluation judges (GPT-4.1 and Qwen2.5-VL-72B replacing CLIP-based metrics for nuanced alignment assessment), and implicitly as the target of improvement via structured training signals that teach compositional reasoning. The GCoT framework represents a T2I analogue of chain-of-thought prompting in language models, raising questions about how well VLMs can supervise generative reasoning tasks they cannot directly generate. PRISM-Bench's finding that even top closed-source T2I models score only ~74 on Text rendering and ~81 on Long Text highlights that the vision–language alignment bottleneck in generation remains far from solved, making this benchmark a useful probe for VLM-driven evaluation research.