Skip to content

LLM-as-Code Agentic Programming for Agent Harness

🕒 Published (v1): 2026-06-14 15:47 UTC · Source: Arxiv · Venue: KDD 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper argues that LLM-as-Orchestrator is a category error — assigning deterministic control flow to a probabilistic system — and proposes Agentic Programming ("LLM-as-Code"), where regular code owns all looping/branching/sequencing and the LLM is invoked only as a callable component for reasoning or generation. A DAG-structured context bounds per-call context to \(O(\text{depth})\) instead of \(O(\text{steps})\), and a GUI agent built on this design reaches 86.8% on OSWorld in 15 steps, outperforming baselines that use 100 steps.

Problem

Mainstream frameworks (ReAct, AutoGen, OpenHands, MetaGPT) make the LLM the orchestrator, which produces three compounding failure modes on long-horizon tasks: (1) token explosion — context grows monotonically as \(O(\text{steps} \times \text{avg\_output})\); (2) control-flow hallucination — the model reports completion before verifier steps run; (3) unreliable completion — correct earlier diagnoses are abandoned because no deterministic mechanism ensures all required steps execute. Better prompts or stronger models reduce per-step error rate without bounding error over a long run; the failures are architectural, not implementation bugs.

Method

Agentic Programming separates the deterministic and probabilistic layers explicitly:

  • Code-driven workflow: The agent's workflow is written as ordinary code. LLM calls are function calls (agent_call(...)) invoked at nodes where reasoning is needed; the runtime (a call-site decorator) enforces looping, branching, and sequencing unconditionally.
  • DAG-structured context: Execution history is the call graph — a DAG of function and reasoning nodes. Each running call sees its full ancestor chain; completed subtrees collapse to a one-line summary before the parent accumulates them. Context length at any call is \(O(\text{depth})\), not \(O(\text{steps})\). The full DAG is retained at the harness level for replay and debugging.
  • Multi-agent collaboration: Agents are functions; parallelism is standard concurrent function invocation. Coordination is the graph structure itself, not a supervisory model. Inter-agent interfaces are typed return values, so failures are localized failed calls the program can retry or reroute.
  • Self-programmed evolution: An agentic function can generate and test other agentic functions; proposed functions are committed as code only after passing tests, making improvements durable rather than re-sampled each run.

Key Contributions

  • Formal diagnosis of LLM-as-Orchestrator as a deterministic/probabilistic category error, unifying token explosion, hallucinated control flow, and unguaranteed compliance under one root cause.
  • The LLM-as-Code abstraction: LLM becomes a typed, callable adaptive component inside an otherwise ordinary program.
  • DAG-structured context scoping rule: live context = ancestor chain only; returned frames collapse to summaries, bounding context by call depth.
  • Demonstration on OSWorld GUI automation: 86.8% overall success in 15 steps vs. 80.4% (best baseline, Holo3-35B-A3B) in 100 steps using the same Claude Sonnet 4.6 model.
  • Analysis showing multi-agent fan-out, failure isolation, and self-programmed evolution fall out of the paradigm without additional mechanisms.

Results

  • OSWorld GUI automation (overall success %):
  • Holo3-35B-A3B (100 steps): 80.4%
  • OpenAPA w/ Gemini-3.1-pro (100 steps): 78.3%
  • Claude Sonnet 4.6 standalone (100 steps): 72.1%
  • LLM-as-Code w/ Claude Sonnet 4.6 (15 steps): 86.8%
  • The same underlying model (Claude Sonnet 4.6) jumps from 72.1% → 86.8% purely from the control-flow architecture change, using 6.7× fewer steps.

Limitations

  • Scope is explicitly restricted to structured workflows; the authors concede fully exploratory tasks (open-ended brainstorming, unstructured research) may still benefit from LLM-driven orchestration.
  • Only one benchmark (OSWorld) is reported; no ablations separating the contribution of DAG context from code-driven control flow are presented.
  • The paper is a KDD 2026 workshop position paper — theoretical claims about compliance guarantees and context bounds are argued analytically rather than via controlled experiments on context length or compliance rate.
  • Self-programmed evolution is described architecturally but not evaluated empirically.
  • Implementation details (call-site decorator, runtime) are deferred to appendices not included in the provided text.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct theoretical and empirical grounding for the harness pattern: it names the construct explicitly ("agent harness") and provides a principled argument for why the harness — not the LLM — must own control flow. The DAG-structured context model is a concrete design for how harnesses should manage state across recursive agent calls without accumulating a flat transcript. The self-programmed evolution component describes a meta-harness capability — a harness that can rewrite and extend its own agentic functions — which is directly relevant to self-improving or auto-configured pipeline designs. The OSWorld result quantifies the reliability gain from moving orchestration out of the model and into the harness layer.