AFlow: Automating Agentic Workflow Generation

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AFlow reformulates agentic workflow construction as a search problem over code-represented LLM-invoking graphs and solves it with a Monte Carlo Tree Search (MCTS) variant in which an LLM optimizer iteratively proposes, evaluates, and backpropagates workflow modifications. It achieves a 5.7% average improvement over manually designed baselines and 19.5% over the best prior automated method (ADAS) across six benchmarks, while enabling smaller models to outperform GPT-4o at 4.55% of its inference cost.

Problem

Constructing effective agentic workflows requires significant human effort, which limits scalability. Prior automated approaches are partial: DSPy/prompt-optimization methods require a manually fixed workflow skeleton; GPTSwarm's graph-RL formulation cannot express conditional logic; ADAS uses code-represented workflows but relies on a linear heuristic search that accumulates experience in a flat list, losing signal as context grows and failing to escape local optima within a limited iteration budget.

Method

AFlow defines a workflow \(W\) as a set of LLM-invoking nodes \(\{N_i\}\), each parameterized by model \(M\), prompt \(P\), temperature \(\tau\), and output format \(F\), connected by code-represented edges \(E\) (supporting sequences, conditionals, loops). The search space is

\[S_{\text{AFlow}} = \{(P_1,\dots,P_n, E, O_1,\dots,O_n) \mid P_i \in \mathcal{P},\, E \in \mathcal{E},\, O_i \in \mathcal{O}\}\]

where \(\mathcal{O}\) is a library of reusable operators — composite node patterns like Generate, Ensemble, Review&Revise, Test, Programmer, and Custom.

MCTS navigates this space in four iterated phases:

1. Soft mixed-probability selection — draws a parent workflow from the top-\(k\) pool using a mixed distribution \(P_{\text{mixed}}(i) = \lambda \cdot \frac{1}{n} + (1-\lambda) \cdot \frac{e^{\alpha(s_i - s_{\max})}}{\sum_j e^{\alpha(s_j - s_{\max})}}\) (\(\lambda=0.2\), \(\alpha=0.4\)), with the blank template always eligible to prevent local optima.
2. LLM-based expansion — the optimizer (Claude 3.5 Sonnet) reads the parent workflow's code plus its full modification/failure history and proposes a new workflow by editing prompts or node connections.
3. Execution evaluation — the candidate workflow runs 5 times on a held-out validation set (20% of data, subsampled to high-variance problems); mean and standard deviation are recorded.
4. Experience backpropagation — modification diffs, improvement/failure outcomes, and prediction logs are stored in the parent tree node for future expansion prompts.

Key Contributions

• Formal unification of workflow optimization as \(W^* = \arg\max_{W \in S} G(W, T)\), generalizing prompt optimization and hyperparameter tuning as special cases.
• AFlow: first fully automated workflow discovery method based on MCTS with tree-structured experience, requiring no manually pre-specified workflow skeleton.
• Operator abstraction as composable building blocks that shrink effective search space and transfer across tasks.
• Empirical demonstration that auto-generated workflows transfer cost efficiency: smaller executor LLMs can beat GPT-4o.

Results

• AFlow (GPT-4o-mini executor) achieves 80.3% average across HotpotQA, DROP, HumanEval, MBPP, GSM8K, MATH-lv5.
• +5.7% over best manually designed baseline (CoT-SC, MedPrompt) averaged across six benchmarks.
• +19.5% over ADAS (best prior automated method) on average.
• +57% over ADAS specifically on MATH-lv5 and MBPP (most complex tasks).
• DeepSeek-V2.5 with AFlow-generated workflow outperforms GPT-4o at 4.55% of GPT-4o's inference cost.
• All baselines tested on GPT-4o-mini; results averaged over 3 runs.

Limitations

• Evaluation functions must be numerical/explicit; no demonstrated extension to open-ended generation tasks lacking programmatic metrics.
• Fixed model \(M\), temperature \(\tau\), and output format \(F\) during search — node-level hyperparameters are excluded from the search space to manage cost.
• 20-iteration budget; convergence is task- and model-dependent.
• Validation set is only 20% of data, sub-sampled further to high-variance problems — potential mismatch with full test distribution.
• Per-iteration cost is non-trivial (5 evaluation runs × validation set); total optimization cost is not fully characterized relative to gains.

Relevance to Harnesses / Meta-Harnesses

AFlow is a direct instantiation of a meta-harness: it takes a task specification and an evaluation function and autonomously synthesizes a task-specific execution harness (agentic workflow) without human scaffolding. The MCTS loop with tree-structured experience is a concrete design pattern for meta-harness search engines — analogous to how a CI meta-harness might auto-tune pipeline stages. The operator library mirrors the "reusable harness primitives" concept: composable, parameterized building blocks that a search process assembles rather than a human configuring by hand. For researchers building harnesses that must generalize across tasks or domains, AFlow's formulation — search space \(= f\)(operators, code-edges, prompts) plus an MCTS optimizer — is a transferable blueprint, and the 19.5% gap over linear-search baselines (ADAS) quantifies the value of structured search over naïve accumulation.