CurateEvo: Data-Curation Evolving for Agentic Post-Training

🕒 Published (v1): 2026-07-07 11:07 UTC · Source: Arxiv · link

Why this paper was selected

Dynamic data curation evolving during agentic post-training; fixes fixed-preprocessing bottleneck

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CurateEvo treats agentic post-training data curation as an executable program that is iteratively rewritten by an LLM-based code-evolution agent using failed trajectories from a held-out dev set. At each evolution epoch the curation code transforms a fixed raw corpus into SFT data, RL data, and an inference-time memory bank, then is revised to improve effectiveness and reduce training cost. On ACEBench-Agent, BFCL-V4, and τ²-Bench it outperforms prior curation baselines by 3.2 points (labeled data) and 2.7 points (wild data).

Problem

Existing agentic post-training pipelines suffer from two failure modes: Curation Neglect — they augment data but ignore filtering and refinement, mixing low-quality trajectories into training — and Adaptation Rigidity — fixed preprocessing pipelines cannot adapt when the trained agent fails on new or out-of-distribution tasks. Neither limitation is addressed by simply collecting more environment interactions.

Method

CurateEvo represents the curation strategy \(\rho\) as executable code that maps a fixed raw corpus \(D_\text{raw}\) to three resources:

\[\rho : D_\text{raw} \to D^\text{sft}_\rho, D^\text{rl}_\rho, M^\text{mem}_\rho\]

The agent policy is then trained from a fixed base model with a fixed SFT+GRPO recipe: \(\pi_\rho = \text{TrainSFT+GRPO}(\pi_\text{base}, D^\text{sft}_\rho, D^\text{rl}_\rho)\), and uses retrieved memory at inference time.

Each epoch proceeds as: (1) execute \(\rho_e\) on \(D_\text{raw}\), (2) train \(\pi_e\), (3) evaluate on held-out dev set \(Q_\text{dev}\), (4) collect failed trajectories \(T^-_e\). A code-evolution agent (GPT-5.4 + mini-SWE-agent) then revises \(\rho_e\) to maximize:

\[J(\rho) = P(\rho) - \lambda C(\rho)\]

where \(P(\rho)\) is dev performance and \(C(\rho) = \tanh\!\left(\frac{\log(1 + \text{Cost}_\text{train}(\rho))}{\log(1 + N_\text{ref})}\right)\) is the normalized training-turn cost. The revision is ordered: an effectiveness pass first diagnoses recurring failure modes (wrong tool selection, invalid argument grounding, poor recovery) and rewrites \(\rho\) to augment, filter, or refine data covering those modes; an efficiency pass then removes redundant trajectories, prunes low-utility turns, and truncates overly long interactions. Revisions that do not improve \(J(\rho)\) are rolled back. Experiments use Qwen3-4B as base policy, \(\lambda=0.3\), \(N_\text{ref}=10^5\), LoRA (\(r=16\)), and 3 evolution epochs.

Key Contributions

• Curation-as-code: curation strategy is executable Python that a downstream LLM rewrites, enabling arbitrary combinations of augmentation, filtering, and refinement.
• Failure-driven evolution loop: dev-set failures serve as the gradient signal for rewriting curation code rather than for directly updating the policy.
• Unified three-resource curation: jointly produces SFT data, RL data, and an inference-time memory bank in one framework.
• Cost-aware efficiency optimization: explicit turn-count cost term removes redundant supervision while preserving coverage of rare failure modes.
• Recipe-agnostic: evolved curation resources are compatible with GRPO, AgentGym-RL, and ProRL-Agent without modification.

Results

• CurateEvo (Qwen3-4B) achieves 56.7 / 52.4 / 41.9 on ACEBench-Agent / BFCL-V4 / τ²-Bench (labeled); 55.8 / 50.0 / 37.2 (wild), outperforming all baselines in both settings.
• Improves over the strongest prior baseline by +3.2 average (labeled) and +2.7 average (wild) across the three benchmarks.
• CurateEvo with Qwen3-4B outperforms MUA-RL and EnvScaler running Qwen3-8B on both labeled and wild data.
• Combined with ProRL-Agent (recipe compatibility experiment): average gain of +21.3 points over ProRL-Agent without CurateEvo data.
• Curation overhead: 0.51M tokens and 405 s per 1K retained training turns, approximately 48% fewer tokens and 50% less wall-clock time than the average prior baseline (range: 0.72M–1.21M tokens, 590–1040 s).
• Ablation: removing effectiveness-oriented revision causes an average drop of ~7 points; removing efficiency-oriented revision causes a smaller but consistent drop; removing any one of SFT data, RL data, or memory each degrades performance.

Limitations

• The code-evolution agent relies on GPT-5.4 + mini-SWE-agent, introducing an external proprietary model dependency and per-epoch compute overhead.
• Evolution runs for only 3 epochs; convergence behavior beyond this (Figure 4 is truncated) is not fully reported.
• Experiments use Qwen3-4B as the base policy; generalization to substantially larger or different-architecture base models is not demonstrated.
• Quality of the held-out dev set constrains the evolution signal; if the dev set is unrepresentative, the curation code may overfit to its failure modes.
• Wild-data setting still yields lower absolute scores than labeled data, indicating that noisy real-user trajectories remain harder to exploit even with adaptive curation.

Relevance to Agentic AI / LLM Agents

CurateEvo addresses a core bottleneck in agentic post-training: the gap between raw trajectory corpora and the high-quality, targeted data needed for long-horizon decision-making. The idea of representing and evolving curation logic as code, driven by agent failures rather than human heuristics, is a natural extension of the broader "LLM-as-optimizer" paradigm to the data pipeline itself. Its three-resource output (SFT + RL + retrieval memory) maps directly to the standard components of modern agent training stacks, making the framework a drop-in complement to methods like GRPO or ProRL. For researchers tracking agentic post-training, this work establishes failure-driven curation evolution as a distinct and practical lever — orthogonal to policy optimization — with measurable efficiency gains.

🔗 Related in this collection

• AgentGym-RL: An Open-Source Framework to Train LLM Agents for Long-Horizon Decision Making via Multi-Turn RL