Skip to content

Tool-Making and Self-Evolving LLM Agents in Low-Latency Systems

🕒 Published (v1): 2026-07-09 00:27 UTC · Source: Arxiv · link

Why this paper was selected

Self-evolving agents compile SOP steps into reusable tools, cutting per-request latency

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Production LLM agents using CodeAct-style inference-time code generation waste latency and introduce run-to-run variance by rediscovering schemas and regenerating code on every request. This paper replaces that loop with an offline agentic tool-making pipeline that compiles repeated SOP steps into validated, versioned Python tools before deployment. Applied to a Fulfillment Center alarm-triage system, the approach reduces p50 latency by 42–62% and end-to-end error rate by up to 53%.

Problem

In CodeAct-style agents, every request re-interprets the same SOP text, re-queries the same MCP backend schema, and regenerates functionally identical code—burning latency, tokens, and introducing non-deterministic variance. For latency-sensitive industrial workflows (e.g., robotic fulfillment center alarm triage where failures cascade within minutes), this is unacceptable. Prior SOP-following benchmarks confirm agents still fail at workflow-guided tasks, and even SOP-enhanced multi-agent systems leave substantial accuracy gaps on operational diagnostics.

Method

The system operates in two phases:

Build time (tool-making pipeline): 1. A data-collector sub-agent (the CodeAct baseline) runs against the live MCP backend on 3 labeled training cases per node, producing an execution trace containing query code, observed schema (field names, datatypes, value ranges), and graded verdicts. 2. A tool-maker LLM (GLM-4.7) synthesizes a candidate tool conditioned on the SOP node text, its position in the 44-node decision tree, and the data-collector's trace. The tool has a uniform signature: (warehouse, timestamp, context) → (verdict, observed_value, threshold, explanation). 3. A reflector–tool-maker repair loop (up to 3 rounds) tests the candidate against the full labeled training set, passes failing (input, expected, predicted) triples to the reflector for a short diagnosis, then conditions the tool-maker on that diagnosis for a rewrite. Early termination on full training-set pass.

Inference time: The production alarm-triage agent (a small fine-tuned model, Qwen3 32B or GLM-4.5-Air) calls compiled tools per SOP node—either via sub-agent delegation or directly inline—falling back to CodeAct only when a tool is unavailable or raises an exception. The agent is fine-tuned on a mixture of tool-available (75%, with 25% random tool dropout) and tool-free (25%) trajectories to preserve fallback capability.

Key Contributions

  • An agentic tool-making pipeline grounded in live execution traces and a test–repair loop that achieves 94.5% per-node pass@1 across 44 decision nodes.
  • Demonstration that both the data-collection trace (+9.2pp over SOP-only) and the test-repair loop (+4.6pp) are individually necessary; only the combined SOP+D+R configuration exceeds 90%.
  • Production deployment showing tools reduce p50 latency 42% and enable a simpler direct-call architecture that reduces p50 a further 62% in controlled ablation.
  • End-to-end error rate reduction of 36% (Qwen3 32B) and 53% (GLM-4.5-Air) on 1,500 historical alarms.
  • Evidence that specification quality—not model capability—becomes the binding constraint once grounding is in place: clarifying ambiguous SOP nodes raises pass@1 from 94.5% to 99.9%.
  • Versioned tools with deterministic I/O surface upstream data drift (type mismatches, cumulative-vs-instantaneous metric confusion, format changes) that non-deterministic code generation masks.

Results

  • Tool-making pass@1 (44-node decision tree, GLM-4.7, SOP+D+R): 94.5% vs. 75.7% (SOP text only), 84.9% (+D only), 89.3% (+R only).
  • GLM-5 full config: 93.6%; Qwen3 235B: 81.1%; GLM-4.7 Flash: 65.8%.
  • Specification improvements (Opus, 4 residual nodes): augmenting edge-case training data → 96.6%; clarifying ambiguous SOP → 99.9%.
  • Production latency (Qwen3 32B, sub-agent architecture): p50 reduced by 42%, p99 by 41%; output tokens −58% (14,998→6,225); sub-agent turns −45% (33.7→18.7).
  • Direct-call architecture ablation (GLM-4.5-Air, 3 seeds, 1,500 alarms): p50 −62%, p99 −30% vs. sub-agent-with-tools.
  • End-to-end accuracy (1,500 alarms): Qwen3 32B with tools 98.3% vs. 97.2% baseline (error rate −36%, p<0.01); GLM-4.5-Air with tools 99.2% vs. 98.2% (error rate −53%, p<0.001).
  • Fallback robustness: Qwen3 32B fine-tuned with tools, evaluated with tools disabled: 96.9% accuracy vs. 97.2% without tools (statistically indistinguishable).
  • Tool library generation cost: 800K tokens one-time for 44 nodes; transfers across models without regeneration.

Limitations

  • Evaluated on a single deployment domain (outbound dock alarm triage); generalization to free-form runbooks or other agentic workflows is unestablished.
  • The improvement loop still requires human oversight before promoting new tool versions to production; full automation is not yet demonstrated to be safe.
  • Clarifying underspecified SOP nodes is inherently a human task, as SOP text encodes author intent not capturable from data alone.
  • Creating new tools still benefits from labeled training cases; the label-free variant trails the labeled ceiling by ~3pp.
  • Weak tool-making models (Qwen3 235B, GLM-4.7 Flash) do not improve monotonically with added grounding components, suggesting model capacity gates pipeline effectiveness.

Relevance to Agentic AI / LLM Agents

This paper directly addresses the self-evolving agent paradigm: rather than relying on inference-time code generation, LLMs build and validate their own reusable action space offline, which is a concrete instantiation of the "skill/tool library" direction that has emerged as an alternative to ReAct/CodeAct loops. The key architectural insight—that compact, deterministic tool outputs eliminate the rationale for a hierarchical sub-agent layer entirely—has implications for how multi-agent systems should be designed when the backend API is stable. The result that SOP specification quality, not model capability, becomes the binding constraint once grounding is in place reframes agent reliability as a knowledge engineering problem as much as a modeling one. The versioned-tool + monitoring-agent pattern is also a practical blueprint for production auditability, a recurring gap in deployed agentic systems.