Skip to content

In-the-Flow Agentic System Optimization for Effective Planning and Tool Use

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Optimizing agentic systems in-flow with decoupled policy for planning and tool use

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AGENTFLOW is a trainable agentic framework that coordinates four specialized modules (planner, executor, verifier, generator) through an evolving shared memory and trains its planner on-policy inside the live multi-turn loop. Its core training algorithm, Flow-GRPO, resolves the long-horizon sparse-reward credit-assignment problem by broadcasting a single final-outcome reward to every turn, decomposing multi-turn RL into a sequence of tractable single-turn updates. A 7B-backbone instance outperforms GPT-4o on ten benchmarks spanning search, agentic, math, and science tasks.

Problem

Monolithic tool-integrated reasoning (TIR) models train a single policy under full multi-turn context, causing training instability as horizons lengthen and tool diversity grows, and brittle generalization to unseen tasks. Existing agentic systems decompose work across specialized modules but remain training-free or use offline (off-policy) supervision decoupled from live multi-turn dynamics, causing distribution shift and poor adaptation to cascading tool-feedback.

Method

AGENTFLOW formalizes multi-turn tool use as a finite-horizon MDP with four modules: - Action Planner \(\pi_\theta\) (trainable): generates sub-goal, selects tool \(k \in \mathcal{K}\), and retrieves context from memory, producing \(a^t \sim \pi_\theta(a^t \mid q, \mathcal{K}, M^t)\). - Tool Executor \(E\): invokes the chosen tool, returns observation \(e^t\). - Execution Verifier \(V\): produces binary signal \(v^t \in \{0,1\}\); \(v^t=1\) terminates the loop. - Solution Generator \(G\): produces final answer \(o \sim G(o \mid q, M^T)\).

Memory \(M^t\) is updated deterministically via \(M^{t+1} = f_\text{mem}(M^t, a^t, e^t, v^t)\), bounding context growth and providing a transparent, structured reasoning record.

Flow-GRPO trains the planner on-policy within the live system. Every action \(a^t\) in a trajectory receives an identical final-outcome reward: $\(r = R(a^t) = \bar{R}(o, q, y^*), \quad \forall t = 1, \ldots, T\)$ where \(\bar{R} \in \{0,1\}\) is assigned by an LLM-as-judge rubric. This broadcasts trajectory-level success uniformly, converting multi-turn RL into independent single-turn updates. The objective averages clipped importance-ratio losses (PPO-style) across \(G\) rollouts, \(T_i\) turns, and token positions, with group-normalized advantages: $\(A^t_i = \frac{\bar{R}(o_i, q, y^*) - \text{mean}_k\{\bar{R}(o_k, q, y^*)\}}{\text{std}_k\{\bar{R}(o_k, q, y^*)\}}\)$

Key Contributions

  • AGENTFLOW framework: four-module agentic system with an evolving structured memory that enables adaptive long-horizon planning while bounding context growth.
  • Flow-GRPO: on-policy RL algorithm for agentic systems that broadcasts a single verifiable outcome reward to all turns, formally reducing multi-turn credit assignment to single-turn policy updates (convergence proof provided).
  • Empirical superiority: 7B-scale AGENTFLOW surpasses all 7B tool-integrated RL baselines and the ~200B GPT-4o across ten diverse benchmarks, with analysis confirming in-the-flow optimization is the causal driver.

Results

  • Search tasks (Bamboogle, 2Wiki, HotpotQA, Musique): AGENTFLOW (w/ Flow-GRPO) avg. 57.3% vs. best 7B tool-integrated baseline VerlTool 39.0% (+14.9%), vs. GPT-4o 49.1% (+8.2%).
  • Agentic reasoning (GAIA): 33.1% vs. Search-R1 19.1% (+14.0%), vs. GPT-4o 17.3% (+15.8%).
  • Mathematical reasoning (AIME24, AMC23, GameOf24): avg. 51.5% vs. ToRL 37.0% (+14.5%); AIME24 40.0% vs. GPT-4o 13.3%.
  • Scientific reasoning (GPQA, MedQA): avg. 63.5% vs. TIR 59.4% (+4.1%).
  • Flow-GRPO ablation: adding Flow-GRPO to the base AGENTFLOW system yields +10.1% on search avg. (47.2%→57.3%) and +19.8% on math avg. (31.7%→51.5%).
  • Positive scaling confirmed with both backbone model size and turn budget.

Limitations

  • Only the planner is trained; executor, verifier, and generator remain frozen, limiting end-to-end joint optimization.
  • Training data mixes only search (Search-R1) and math (DeepMath) domains; generalization to other tool types not demonstrated during training.
  • Evaluation of GAIA uses the textual split only, which may not reflect the full multimodal difficulty of the benchmark.
  • The LLM-as-judge reward model introduces potential noise into the binary reward signal \(\bar{R}\).
  • Computational cost of running the full four-module system at rollout time is not analyzed relative to monolithic TIR approaches.

Relevance to Agentic AI / LLM Agents

AGENTFLOW directly advances the core open problem of trainable multi-module agentic systems: prior work forced a choice between flexible modular architectures (training-free) and trainable models (monolithic). Flow-GRPO's insight—broadcasting trajectory-level reward to every turn to sidestep multi-turn credit assignment—is a clean, broadly applicable algorithm that could generalize to any agentic framework with a differentiable planning policy. The result that a 7B on-policy trained planner beats ~200B proprietary models challenges assumptions about parameter scale as the primary lever for agentic capability, shifting focus to training-time alignment with live tool-use dynamics. This sits at the intersection of RL for LLMs (GRPO/RLVR) and multi-agent orchestration, making it foundational reading for anyone building or training tool-use agents.