Learning to Construct Practical Agentic Systems¶

🕒 Published (v1): 2026-05-29 16:02 UTC · Source: Arxiv · link

Why this paper was selected

Empirical study exposing gap between research and production agentic system requirements

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper proposes a lightweight agentic framework built around "pseudo-tools" (ptools)—LLM calls with restricted context that act as typed, modular subroutines—and shows that hand-constructed static workflows using them outperform dynamic ReAct loops in both accuracy and cost. It then introduces learning methods to induce ptools from agent traces, distill ptools to Python code, and learn/hill-climb workflows, culminating in NSGA-II-based bi-objective optimization over cost and quality.

Problem¶

Production agentic systems prioritize simplicity, controllability, and cost predictability, yet most research focuses on dynamic (ReAct-style) planning that is unpredictable in cost and fragile in practice. There is also no systematic study of hand-engineered vs. learned static workflows across a diverse benchmark suite using a common experimental harness.

Method¶

The core abstraction is an interface—a typed Python function stub with a natural-language docstring—that can be bound to alternative implementations: raw Python, a single LLM call, a ReAct sub-agent, or on-the-fly generated code. Pseudo-tools are interfaces bound to LLM calls with restricted context, enabling modular reasoning without the full attention context bleed of a monolithic agent loop.

Ptool induction runs a four-stage LLM pipeline over recorded rollouts: (1) label each reasoning step with an action type, (2) merge synonymous labels, (3) threshold by frequency, (4) synthesize a typed stub per retained category. The result is a Python module of K ptools, each backed by a single LLM call at runtime.

Code distillation replaces individual ptool LLM calls with generated Python functions: a strong code-generating LLM proposes $n$ candidate functions, selects the best by training accuracy over $R$ rounds with failure feedback, and retains only those with low holdout commission-error rates (with backoff to the LLM ptool otherwise).

Workflow learning is done two ways: (a) code distillation applied to the root interface, generating an orchestrating Python function from sampled tool-call traces; (b) iterative hill-climbing (orchestration learner) where a supervisor LLM rewrites workflow source and ptool docstrings, accepting edits that improve training accuracy or match it with better eval accuracy, terminating after 5 consecutive non-improvements.

Bi-objective optimization uses NSGA-II (via DEAP) over a chromosome encoding interface-level genes (implementation choice + backend LLM assignment), evaluating configurations on validation data for correctness and cost per query, with LLM-call caching to reduce sweep cost by up to 8×.

Key Contributions¶

Pseudo-tool framework: typed interfaces with swappable implementations enforce modularity by restricting context at subtask boundaries, formalized as a lightweight extension (<1000 lines) of Pydantic AI.
Empirical evidence for static workflows: across 19 diverse benchmarks, hand-engineered static workflows average 0.64 accuracy vs. 0.56 for ReAct, at $0.70 vs. $1.66 per 100 examples, quantifying the "agentic tax."
Ptool induction from traces: four-stage LLM pipeline extracts reusable typed subroutines from ReAct/CoT rollouts; induced ptools outperform human-engineered toolkits on 5/6 tested tasks under the same ReAct loop.
Code distillation for ptools and workflows: replaces LLM calls with zero-cost Python functions when feasible, with safe abstention/backoff.
Hill-climbing orchestration learner: jointly edits workflow source and ptool docstrings over iterations, outperforming hand-engineered baselines (avg. 0.69–0.76 vs. 0.55 for human/human).
NSGA-II bi-objective sweep: simultaneously optimizes output quality and inference cost over discrete configuration space, supporting heterogeneous LLM assignment per interface.

Results¶

Static vs. ReAct (Table 1, 19 benchmarks, DeepSeek-V3.1): static workflows average 0.64 accuracy vs. 0.56 ReAct vs. 0.47 zero-shot; cost $0.70 vs. $1.66 vs. $0.44 per 100 examples.
Learned ptools (ReAct/learned) (Table 2, 8 tasks): average 0.54 vs. 0.44 for ReAct/human and 0.55 for human/human.
Code distillation (CodeDist/human): average 0.67 on selected tasks, reaching 0.97 on NaturalPlan Trip and 1.00 on RuleArena NBA.
Orchestration learner (Orch-ToolSeed/learned): average 0.76, highest reported, outperforming human/human (0.55) and Orch-WfSeed/human (0.62).
Caching reduces NSGA-II sweep cost by up to 8×.

Limitations¶

Ptool induction hyperparameters ($K$, minimum frequency, correct-only flag) are fixed for all experiments; sensitivity is uncharacterized.
Code distillation commission errors require a holdout safety check but cannot be recovered from at runtime if they occur; the abstention mechanism only partially mitigates this.
Benchmark coverage is 19 tasks hand-selected by coauthors, introducing selection bias; the hill-climbing budget is fixed at 5 iterations, potentially underexploring.
The orchestration learner requires recorded rollouts and a training/eval split; cold-start with no existing trajectories relies on a single LLM composition step whose quality is not systematically evaluated.
Cost measurements use Together.ai's DeepSeek-V3.1 pricing; results may not generalize to other cost structures.
The paper is truncated; full bi-objective Pareto frontier results and ablations are not visible.

Relevance to Harnesses / Meta-Harnesses¶

This paper is a direct contribution to the meta-harness design space: it frames the entire agentic construction problem as a configurable harness (interfaces + bindings + configurations serializable to/from YAML) and then builds learning systems on top of that harness to automate workflow and tool construction. The ptool induction pipeline is itself a multi-stage LLM meta-harness that takes traces as input and emits a new harness component (the ptool library). The orchestration learner is similarly a meta-level hill-climber that iteratively rewrites harness components (workflow Python source, ptool prompts, model assignments), directly instantiating the "meta-agent that improves the agent" pattern tracked in this topic. The NSGA-II sweep over configuration genes further operationalizes multi-objective harness optimization as a first-class design concern.