Skip to content

TaskCraft: Automated Generation of Agentic Tasks

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Automated generation of diverse agentic tasks for scalable agent training and evaluation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TaskCraft is an automated workflow for generating scalable, verifiable, multi-tool agentic tasks without human annotation. It progressively complexifies atomic single-tool tasks via depth-based (multi-hop sequential) and width-based (parallel sub-task) extensions, then uses rejection sampling and LLM-based linguistic validation to filter quality. Training on TaskCraft-generated data achieves state-of-the-art on four agentic benchmarks.

Problem

Existing agentic benchmarks (GAIA, BrowseComp, HLE) are annotation-bottlenecked—HLE required 1,000 experts for 2,500 examples—making large-scale agent training data scarce. Prior synthetic data methods (Self-Instruct, WizardLM) target static instruction-following and do not model multi-tool, multi-step agentic execution with verifiable answers.

Method

TaskCraft operates in three stages:

  1. Atomic task generation: Given an unlabeled corpus, a tool is executed on an input index \(i_T\) to yield context \(C\). An LLM samples candidate answers \(a\) from \(C\), infers the relation \(R\) between \(a\) and \(C\), then constructs question \(q = f_q(i_T, R)\). This grounds tasks in actual tool outputs rather than LLM imagination.

  2. Task extension:

  3. Depth-based: An \(n\)-hop task is extended to \((n+1)\)-hop by having a search agent find a superset index \(i_T^{n+1}\) of \(i_T^n\), constructing an intermediate sub-task, then merging: \((q^{n+1}, a) = (f_m(q^n, \hat{q}^{n+1}, i_T^n), a)\).

  4. Width-based: Two independent sub-tasks \(q_1 \to a_1\) and \(q_2 \to a_2\) are merged via LLM into a single task requiring parallel execution: \((q_\text{width} = q_1 + q_2) \to a_1 + a_2\).

  5. Verification: Atomic tasks are validated by checking that a tool-equipped agent solves them but an LLM alone cannot (rejection sampling). Extensions are validated purely through LLM linguistic analysis of incremental changes—no full re-execution—filtering invalid supersets and information leakage.

Bootstrap few-shot prompt optimization (DSPy-style) tunes four critical prompts to improve pass rates (atomic: 54.9% → 68.1%; depth-wise: 41.0% → 51.2%).

Generated trajectories are used for SFT and RL (DAPO) on tool-integrated reasoning (TIR) models with structured <tool>/<observation>/<think> tags.

Key Contributions

  • First automated pipeline for generating multi-tool, multi-hop, verifiable agentic tasks at scale.
  • Depth-based and width-based extension strategies that produce tasks requiring chain-of-tool and parallel-tool reasoning.
  • Incremental verification scheme: agent-based validation only at atomic creation; LLM linguistic analysis for extensions—cutting validation cost without compromising quality.
  • Dataset of ~41k tool-intensive tasks, 12.6k tool-interaction trajectories, 5k multi-hop decompositions across web, PDF, and image modalities.
  • SOTA results on GAIA, WebWalker, BrowserComp, and HLE via SFT + RL on TaskCraft data.

Results

  • GAIA (Qwen-2.5-32B): MHQA baseline 38.8% → 5k MHQA + 2.5k TaskCraft SFT 60.2% (+21.4pp) → +8k TaskCraft RL 60.8% (+22.0pp), SOTA among TIR models.
  • WebWalker (Qwen-2.5-7B): 5k MHQA + 2.5k TaskCraft achieves 59.4%, surpassing QwQ-32B (46.5%) and WebSailor 32B (43.2%).
  • BrowserComp (Qwen-2.5-32B): 24.8% vs. WebSailor's 22.4%.
  • HLE (Qwen-2.5-32B): 20.6% with SFT+RL vs. 20.0% (MHQA only SFT).
  • TaskCraft SFT-only models match SOTA systems requiring both SFT and RL.
  • GAIA scaling: 1k → 3k → 5k tasks yields 17.5% → 31.1% → 39.8% Pass@3 on GAIA-103.
  • Human evaluation: atomic tasks score 95.0% accuracy, 91.7% linguistic fluency, 83.3% single-answer; depth-based extension validity 82.3%.
  • Tool-context structured generation vs. LLM-only: pass rate 43.0% vs. 18.5%, generation time 86.7s vs. 119.7s.

Limitations

  • Width-based extension is described for two sub-tasks only; generalization to \(k > 2\) parallel branches is unaddressed.
  • Validation relies on a judge-LLM, which may itself make errors, particularly for factual or time-sensitive tasks (Figure 5 shows cases where the generated answer is outdated or incorrect).
  • Corpus is 75% web, 15% image, 10% PDF; domain breadth of generated tasks is constrained by seed corpus collection.
  • Depth-based extension requires a live search agent to find superset indices, making generation latency and cost non-trivial at scale.
  • Tasks are English-only; cross-lingual or multilingual agentic evaluation is out of scope.

Relevance to Agentic AI / LLM Agents

TaskCraft directly addresses the data bottleneck for training capable tool-using agents by automating the generation of the hardest class of training examples—multi-hop, multi-tool tasks with verifiable answers. The depth/width extension framework mirrors the compositional reasoning structure that real-world agent tasks exhibit (sequential dependency vs. parallel retrieval), making generated data structurally faithful to deployment conditions rather than merely syntactically varied. The rejection-sampling verification criterion (agent succeeds, LLM alone fails) operationalizes a principled definition of "genuinely agentic" tasks and is directly reusable by other benchmark construction efforts. Combined with DAPO-based RL, the pipeline represents a complete self-improving loop—task generation → trajectory sampling → policy update—that could compound as agent capabilities grow.