Skip to content

Compiling Agentic Workflows into LLM Weights: Near-Frontier Quality at Two Orders of Magnitude Less Cost

🕒 Published (v1): 2026-05-21 13:54 UTC · Source: Arxiv · link

Why this paper was selected

Compiles orchestration workflows into LLM weights; claims 100Ă— cost reduction over frameworks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes "subterranean agents"—small LLMs fine-tuned on synthetically generated workflow conversations—as a replacement for external agent orchestration frameworks (LangGraph, CrewAI, etc.). An 8B compiled model achieves 87–98% of frontier in-context quality at 128–462× lower per-conversation cost, with a recompile cycle of 30–50 minutes on production hardware.

Problem

External orchestration frameworks (LangGraph, CrewAI, Google ADK, etc.) dominate developer adoption despite three fundamental costs: they consume context window capacity by injecting full procedure text every turn, require frontier-model API calls at frontier rates per conversation, and expose proprietary procedures to third-party providers. Prior work on weight-compiled agents (SimpleTOD, FireAct, WorkflowLLM) demonstrated feasibility but was not adopted; this paper identifies and empirically addresses the three perceived barriers—quality, cost, and flexibility—that explain that adoption gap.

Method

The paper defines procedures as directed graphs \(F = (N, E, n_0, T)\) with typed nodes (agent/user roles with prompt templates), conditional edges, a start node, and terminal nodes (success, abandonment, escalation). The compilation pipeline has four stages:

  1. Flowchart definition: encode the procedure as a node-edge graph with branching conditions.
  2. Synthetic data generation: traverse all valid acyclic paths, sample scenario variables, and generate turn-by-turn conversations using Claude Sonnet 4.5 as a generator. Each training example consists only of natural dialogue—no procedural annotations are visible at inference time.
  3. Full-parameter fine-tuning: LoRA/PEFT is explicitly rejected; full parameter updates are required because procedural internalization modifies implicit state-tracking behavior more deeply than stylistic alignment. Qwen 2.5 3B (single RTX 5090) and Qwen3-8B (8Ă—A100 with DeepSpeed ZeRO-3) are the base models.
  4. Runtime deployment: the compiled model receives only a minimal system prompt (e.g., "You are a helpful travel booking assistant") with no flowchart state, routing logic, or procedural instructions injected.

Three domains are evaluated (n = 200 scenarios each): travel booking (14 nodes, 86 paths), Zoom support (14 nodes, 60 paths, product-specific knowledge), and insurance claims (55 nodes, 6 decision hubs, 2,381 paths). Baselines are a LangGraph orchestrator (Claude Sonnet 4.5, ~70× more parameters) and an in-context baseline (full serialized flowchart in system prompt, Claude Sonnet 4.5). Quality is measured via LLM-as-judge (Claude Sonnet 4.5 + independent GPT-4.1 cross-check) on five 1–5 criteria: Task Success, Information Accuracy, Consistency, Graceful Handling, Naturalness. Statistics use Wilcoxon signed-rank or Mann–Whitney U with Holm–Bonferroni correction.

Key Contributions

  • Empirical proof that a compiled 8B model achieves 87–98% of frontier in-context quality across three domains of increasing procedural complexity (14–55 nodes).
  • Same-model controlled comparison showing compilation strictly dominates explicit orchestration on the same 3B base model (4 of 5 criteria, \(p < 0.001\)).
  • Cost analysis: 128–462Ă— cheaper per conversation than the in-context frontier baseline, compounding a ~65Ă— per-token self-hosting reduction with a 2–7Ă— token volume reduction that grows with procedure complexity.
  • Failure rate reduction: compiled model shows 5.5% vs. 24.0% (travel) and 9.0% vs. 17.0% (insurance) task failure compared to LangGraph orchestrator, eliminating routing errors by construction.
  • Recompile cycle characterization: 30–50 min on 8Ă—H200 (or ~3–4 hrs on single A100), establishing this as a CI/CD-scale operation rather than a paradigm shift.
  • Scalability demonstration: compilation holds at 55-node, 2,381-path procedure (insurance claims), the largest tested.

Results

  • Travel (3B): compiled 3B beats same-model 3B orchestrator on task success (+0.18), consistency (+0.22), graceful handling (+0.20), naturalness (+0.17), all \(p < 0.001\). Reaches ~82% of in-context baseline on graceful handling and naturalness.
  • Zoom (8B): closes gap—92% of in-context on graceful handling (vs. 82% for 3B), 97% on naturalness; 87% on information accuracy is the remaining bottleneck. Leads LangGraph orchestrator on naturalness (4.87 vs. 4.64, \(p < 0.001\)).
  • Insurance (8B): 92–98% of in-context quality (4.40–4.92 vs. 4.78–5.00); beats LangGraph on graceful handling (4.81 vs. 4.38, \(p < 0.001\)) and naturalness (4.92 vs. 4.58, \(p < 0.001\)).
  • Cost: $0.0010/conv (travel), $0.0003/conv (Zoom), $0.0007/conv (insurance) vs. $0.133, $0.103, $0.327 for in-context baseline. IC/Sub ratios: 128Ă—, 296Ă—, 462Ă—.
  • Latency: 2.8Ă— faster than LangGraph in insurance (43.2s vs. 120.8s); comparable in travel (69.4s vs. 64.9s).
  • GPT-4.1 cross-judge: confirms 83–99% of in-context quality, qualitative ordering unchanged.

Limitations

  • Evaluation domains are all customer-service procedural tasks; generalization to open-ended agentic tasks (tool use, web browsing, code execution) is not tested.
  • Quality gap on information accuracy persists at the 8B scale (87% for Zoom), attributed to world knowledge capacity rather than procedure-following; not resolved.
  • Full-parameter fine-tuning requirement means LoRA/PEFT cannot be used, imposing hardware minimums (single A100 80GB for 8B).
  • Synthetic training data is generated by Claude Sonnet 4.5; quality of the compiled model is upper-bounded by the generator and may inherit its failure modes.
  • Recompile flexibility assumes access to either a multi-GPU cluster (30–50 min) or patience for a 3–4 hour single-GPU cycle; neither may be available in all deployment contexts.
  • LLM-as-judge methodology carries known self-preference bias; cross-validation with GPT-4.1 mitigates but does not eliminate this.
  • Break-even analysis for compilation cost is application-specific and reaches ~500 conversations; very low-volume deployments may not recoup compilation cost.

Relevance to Harnesses / Meta-Harnesses

This paper directly challenges the prevailing meta-harness paradigm—where an orchestration layer (LangGraph, CrewAI, etc.) wraps LLM calls and manages workflow state at runtime—by demonstrating that the harness itself can be compiled away into model weights. For researchers tracking harnesses and meta-harnesses, this is a fundamental alternative architecture: instead of a harness that interprets a workflow graph at runtime, the graph is consumed during training and the harness ceases to exist at inference time. The "subterranean agent" framing inverts the usual abstraction layer hierarchy: the workflow graph is a build artifact, not a runtime dependency, and the resulting compiled model is a self-contained unit with no external orchestration surface. This raises concrete questions for harness designers about where procedural knowledge should live (weights vs. context vs. orchestration layer) and which tasks are better served by compile-time vs. runtime workflow management.