Skip to content

DataEvolver: Self-Evolving Multi-Agent Data Construction for Text-Rich Image Generation

🕒 Published (v1): 2026-06-30 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

Self-evolving multi-agent data construction — meta-harness pattern for iterative refinement

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DataEvolver is a self-evolving multi-agent framework that reframes text-rich image dataset construction as iterative policy refinement, converting rejected samples into semantic feedback that guides subsequent retrieval and synthesis rounds. At the 0.75M scale on PixArt-α, it improves OCR-F1 by 85.3% on TextScenesHQ and 35.3% on LongTextBench over the strongest static-pipeline baseline. Gains transfer to a second downstream generator (Show-o2), confirming the approach is model-agnostic.

Problem

Existing text-rich image data pipelines follow a crawl–filter–freeze paradigm: candidates are collected, filtered once, and rejected samples are discarded. This wastes the diagnostic signal embedded in failures (OCR errors, duplicate queries, under-covered topics, semantic mismatches) and causes recurring failure modes across construction rounds with no mechanism to correct them.

Method

DataEvolver treats data construction as iterative policy refinement over a policy \(\pi_t = (Q_t, P_t, E_t)\), where \(Q_t\) is the retrieval query policy, \(P_t\) is the generation-prompt policy, and \(E_t\) is an experience library. Four cooperative agents form a closed loop:

  1. Retriever — generates structured, policy-conditioned query templates to build a topic-aware candidate pool from external sources.
  2. Verifier — applies OCR (PaddleOCR), perceptual deduplication (pHash), semantic consistency (CLIP ViT-B/32), and image quality scoring to produce a pass set \(I^t_\text{pass} \subseteq I^t_\text{raw}\) and a round-level feedback signal \(s_t = (\rho_t, \bar{O}_t, \bar{C}_t, \bar{I}_t, R_t)\) capturing pass rate, mean OCR quality, semantic consistency, image quality, and rejection-cause distribution.
  3. Critic — an LLM (Qwen3.5-4B) that maps \(s_t\) into natural-language semantic feedback \(F_t = \text{Critic}(s_t, s_{t-1}, E_{t-1})\), driving query diversification, prompt refinement, and experience-library updates for the next-round policy \(\pi_{t+1}\).
  4. Generator — identifies under-covered topic–subtopic pairs via coverage count \(G^t(u,v) = |\{i \in I^t_\text{pass}: \text{topic}(i)=u, \text{subtopic}(i)=v\}|\) and synthesizes targeted images (Qwen-Image) for deficient regions; generated samples pass through the same fixed Verifier before admission.

Policy revision operates entirely in natural-language space—no gradient updates to any model.

Key Contributions

  • Reformulation of dataset construction as failure-aware construction policy evolution, where rejected samples supply actionable feedback rather than being discarded.
  • DataEvolver: a closed-loop four-agent framework (Retriever → Verifier → Critic → Generator) that converts round-level rejection statistics into semantic policy updates across construction rounds.
  • Demonstration that OCR-F1 improvements scale with data budget and transfer across downstream generators (PixArt-α and Show-o2).
  • Ablation evidence isolating the contribution of the Critic (policy revision) and Generator (coverage completion) as individually necessary components.

Results

  • PixArt-α, TextScenesHQ (0.75M): DataEvolver OCR-F1 = 8.45 vs. MARIO 4.56 vs. AnyWord 0.63 — +85.3% over strongest baseline.
  • PixArt-α, LongTextBench (0.75M): OCR-F1 = 9.08 vs. MARIO 6.71 vs. AnyWord 0.54 — +35.3% over strongest baseline.
  • Show-o2, TextScenesHQ (0.75M): F1 = 0.45 vs. MARIO 0.19; Show-o2, LongTextBench: F1 = 0.44 vs. MARIO 0.27.
  • DataEvolver also achieves lowest FID on TextScenesHQ for both downstream models when reference images are available.
  • Ablation (PixArt-α, 0.1M, TextScenesHQ): removing Critic drops F1 from 1.78 → 1.01; removing Generator drops F1 to 1.40; both components are individually necessary.
  • Process-level analysis: Critic-guided query diversification reduced deduplication failures from 278 to 148 in one logged example round.

Limitations

  • OCR-F1 values are low in absolute terms even after improvement (e.g., 8.45% on TextScenesHQ), indicating text-rich image generation remains very hard and the data gains alone are insufficient for high accuracy.
  • CLIP Score does not consistently improve alongside OCR-F1, suggesting the framework optimizes text legibility at some cost to global semantic alignment.
  • LLM agents used for policy revision (Mistral-7B, Qwen3.5-4B) are modest in scale; stronger critic models may yield larger policy-update quality gains not explored here.
  • Verifier criteria are fixed throughout; the paper explicitly excludes the Verifier from the optimized policy variables, leaving adaptive verification as future work.
  • Evaluation is limited to two benchmarks (TextScenesHQ, LongTextBench) and two downstream generators; broader generalization is unverified.
  • The approach requires multiple construction rounds and LLM inference for policy revision, increasing construction-time compute relative to static pipelines.

Relevance to Harnesses / Meta-Harnesses

DataEvolver is a direct implementation of the self-improving meta-harness pattern: the orchestration layer evaluates each generation round's output (via OCR quality signals), converts rejected samples into structured semantic feedback, and uses that signal to update retrieval and synthesis policy for the next round — closing the loop without human intervention. This is the same core architecture as agent harnesses that self-correct via execution feedback, but applied to the data pipeline itself rather than task completion, making dataset construction the target of the optimization loop. The model-agnostic transfer result (PixArt-α → Show-o2) validates that harness logic and downstream model are decoupled, which is the property you want when designing a harness that should work across generator backends. For practitioners building agentic pipelines, this paper concretizes how to wire rejection signals from a critic agent back into upstream retrieval policy — a reusable scaffolding pattern for any iterative multi-agent data or annotation loop.