Skip to content

DataEvolver: Self-Evolving Multi-Agent Data Construction for Text-Rich Image Generation

🕒 Published (v1): 2026-06-30 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DataEvolver is a self-evolving multi-agent framework that reframes text-rich image dataset construction as iterative policy refinement, converting rejected samples into semantic feedback that guides subsequent retrieval and synthesis rounds. At the 0.75M scale on PixArt-α, it improves OCR-F1 by 85.3% on TextScenesHQ and 35.3% on LongTextBench over the strongest static-pipeline baseline. Gains transfer to a second downstream generator (Show-o2), confirming the approach is model-agnostic.

Problem

Existing text-rich image data pipelines follow a crawl–filter–freeze paradigm: candidates are collected, filtered once, and rejected samples are discarded. This wastes the diagnostic signal embedded in failures (OCR errors, duplicate queries, under-covered topics, semantic mismatches) and causes recurring failure modes across construction rounds with no mechanism to correct them.

Method

DataEvolver treats data construction as iterative policy refinement over a policy \(\pi_t = (Q_t, P_t, E_t)\), where \(Q_t\) is the retrieval query policy, \(P_t\) is the generation-prompt policy, and \(E_t\) is an experience library. Four cooperative agents form a closed loop:

  1. Retriever — generates structured, policy-conditioned query templates to build a topic-aware candidate pool from external sources.
  2. Verifier — applies OCR (PaddleOCR), perceptual deduplication (pHash), semantic consistency (CLIP ViT-B/32), and image quality scoring to produce a pass set \(I^t_\text{pass} \subseteq I^t_\text{raw}\) and a round-level feedback signal \(s_t = (\rho_t, \bar{O}_t, \bar{C}_t, \bar{I}_t, R_t)\) capturing pass rate, mean OCR quality, semantic consistency, image quality, and rejection-cause distribution.
  3. Critic — an LLM (Qwen3.5-4B) that maps \(s_t\) into natural-language semantic feedback \(F_t = \text{Critic}(s_t, s_{t-1}, E_{t-1})\), driving query diversification, prompt refinement, and experience-library updates for the next-round policy \(\pi_{t+1}\).
  4. Generator — identifies under-covered topic–subtopic pairs via coverage count \(G^t(u,v) = |\{i \in I^t_\text{pass}: \text{topic}(i)=u, \text{subtopic}(i)=v\}|\) and synthesizes targeted images (Qwen-Image) for deficient regions; generated samples pass through the same fixed Verifier before admission.

Policy revision operates entirely in natural-language space—no gradient updates to any model.

Key Contributions

  • Reformulation of dataset construction as failure-aware construction policy evolution, where rejected samples supply actionable feedback rather than being discarded.
  • DataEvolver: a closed-loop four-agent framework (Retriever → Verifier → Critic → Generator) that converts round-level rejection statistics into semantic policy updates across construction rounds.
  • Demonstration that OCR-F1 improvements scale with data budget and transfer across downstream generators (PixArt-α and Show-o2).
  • Ablation evidence isolating the contribution of the Critic (policy revision) and Generator (coverage completion) as individually necessary components.

Results

  • PixArt-α, TextScenesHQ (0.75M): DataEvolver OCR-F1 = 8.45 vs. MARIO 4.56 vs. AnyWord 0.63 — +85.3% over strongest baseline.
  • PixArt-α, LongTextBench (0.75M): OCR-F1 = 9.08 vs. MARIO 6.71 vs. AnyWord 0.54 — +35.3% over strongest baseline.
  • Show-o2, TextScenesHQ (0.75M): F1 = 0.45 vs. MARIO 0.19; Show-o2, LongTextBench: F1 = 0.44 vs. MARIO 0.27.
  • DataEvolver also achieves lowest FID on TextScenesHQ for both downstream models when reference images are available.
  • Ablation (PixArt-α, 0.1M, TextScenesHQ): removing Critic drops F1 from 1.78 → 1.01; removing Generator drops F1 to 1.40; both components are individually necessary.
  • Process-level analysis: Critic-guided query diversification reduced deduplication failures from 278 to 148 in one logged example round.

Limitations

  • OCR-F1 values are low in absolute terms even after improvement (e.g., 8.45% on TextScenesHQ), indicating text-rich image generation remains very hard and the data gains alone are insufficient for high accuracy.
  • CLIP Score does not consistently improve alongside OCR-F1, suggesting the framework optimizes text legibility at some cost to global semantic alignment.
  • LLM agents used for policy revision (Mistral-7B, Qwen3.5-4B) are modest in scale; stronger critic models may yield larger policy-update quality gains not explored here.
  • Verifier criteria are fixed throughout; the paper explicitly excludes the Verifier from the optimized policy variables, leaving adaptive verification as future work.
  • Evaluation is limited to two benchmarks (TextScenesHQ, LongTextBench) and two downstream generators; broader generalization is unverified.
  • The approach requires multiple construction rounds and LLM inference for policy revision, increasing construction-time compute relative to static pipelines.

Relevance to Agentic AI / LLM Agents

DataEvolver is a concrete instantiation of a feedback-driven multi-agent loop applied to data curation rather than task execution, demonstrating that agentic architectures with memory (experience library), critique, and tool use (retrieval + synthesis) generalize beyond conversational or reasoning tasks. The Critic agent's role—converting unstructured failure statistics into natural-language policy revisions stored in memory across rounds—mirrors Reflexion-style self-improvement but applied to a data pipeline rather than a reasoning trace. For researchers tracking agentic AI, this paper shows that closed-loop agent systems can govern large-scale infrastructure tasks (dataset construction at 0.75M samples), with quantifiable downstream benefits that ablation ties specifically to the agent roles. It also raises design questions about when to use LLM-mediated natural-language policy updates versus gradient-based optimization, relevant to the emerging literature on LLMs as optimizers and meta-learners.