Skip to content

Code as Agent Harness

🕒 Published (v1): 2026-05-18 17:59 UTC · Source: Arxiv · link

Why this paper was selected

Establishes code as executable agent harness; foundational paradigm shift for agentic systems

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This survey formalizes code as agent harness: the view that code is not merely an LLM output but the executable, inspectable, and stateful substrate through which agents reason, act, model their environments, and coordinate with other agents. It organizes the literature into three connected layers—harness interface, harness mechanisms, and multi-agent scaling—and maps these onto five application domains. The paper argues the bottleneck of agentic autonomy is harness reliability, not just base-model capability.

Problem

Existing surveys treat code as an end product generated by LLMs. As LLMs are embedded in long-running agentic systems, code increasingly serves as the operational medium connecting model outputs to external execution, persistent state, and verifiable feedback—but no unified framework characterizes this role. The gap is a lack of a systematic, layered account of how code functions as infrastructure (interface, mechanism, and multi-agent substrate) rather than as artifact.

Method

The paper is a structured survey organized around a three-layer taxonomy of code as agent harness:

  1. Harness Interface (§2): Three roles of code in the agent loop:
  2. Code for Reasoning: program-delegated reasoning (PoT, PAL, Chain-of-Code), formal/symbolic verification, iterative code-grounded reasoning—externalizing computation so the harness can verify intermediate states.
  3. Code for Acting: grounded skill selection, programmatic policy generation (Code-as-Policies, Voyager), lifelong code-based agents that accumulate skill libraries.

  4. Code for Environment Modeling: structured world representations (repositories, DOM trees), execution-trace world modeling, code-grounded evaluation environments, verifiable environment construction.

  5. Harness Mechanisms (§3): What keeps a code-harnessed agent reliable across steps:

  6. Planning: linear decomposition, structure-grounded, search-based, orchestration-based.
  7. Memory: working, semantic, experiential, long-term, multi-agent, and context-compaction memory.
  8. Tool Use: function-oriented, environment-interaction, verification-driven, workflow-orchestration.
  9. Harness Control: a plan–execute–verify loop using sandboxed execution, permissioned state transitions, and deterministic sensors.

  10. Adaptive Harness Optimization: deep telemetry as optimization substrate, evolution agents, governed harness mutation.

  11. Scaling the Harness (§4): Multi-agent orchestration where shared code artifacts (repositories, tests, traces, workflows) provide a common workspace. Covers agent role specialization (manager/planner/coder/reviewer/tester), collaboration modes (programming, repair, debate, red-teaming), workflow topologies (centralized, distributed, streaming), execution-feedback integration, and shared-harness synchronization including harness-state convergence and transactional conflict resolution.

The survey also distinguishes three conceptual elements: model-internal capabilities, system-provided harness infrastructure, and agent-initiated code artifacts (the underexplored category this work centers).

Key Contributions

  • Formalizes code as agent harness as a conceptual reframe: code transitions from generated artifact to executable operational substrate.
  • Three-layer taxonomy (interface → mechanisms → scaling) that unifies a broad literature under a single harness-engineering lens.
  • Systematic synthesis of agent-initiated code artifacts (regression tests, DSL programs, temporary tools, reusable skills, executable workflows) as a distinct and underexplored category separate from system-provided infrastructure.
  • Cross-domain application mapping: coding assistants, GUI/OS automation, scientific discovery, embodied agents, personalization/recommendation.
  • Articulates six open harness-engineering challenges: evaluation beyond final-task success, verification under incomplete feedback, regression-free harness self-improvement, consistent shared state across agents, human-in-the-loop safety, and multimodal harness extensions.

Results

This is a survey paper; it reports no original empirical benchmarks. The text cites representative systems and results from the surveyed literature (e.g., SWE-Bench, AgentBench, CRUXEval) but does not contribute new numbers. No headline figures from the paper itself to report.

Limitations

  • No original experiments or ablations; claims about relative importance of harness layers are not empirically validated.
  • Coverage is bounded to literature up to 2026 as declared, so recent systems may be absent.
  • The taxonomy boundaries (e.g., what counts as "code" vs. "natural language") are asserted rather than formally justified; edge cases (structured prompts, config files, logs) are acknowledged but not fully resolved.
  • The distinction between agent-initiated code artifacts and system-provided harness infrastructure is conceptually important but operationally blurry in practice—the survey does not provide a crisp operationalization.
  • Open problems are listed but not accompanied by proposed solutions or evaluation criteria.

Relevance to Harnesses / Meta-Harnesses

This paper is the most directly relevant survey in the field for researchers tracking harnesses and meta-harnesses: it introduces harness engineering as a named subdiscipline and provides the most comprehensive layered taxonomy to date. The three-layer model (interface → mechanisms → scaling) is a canonical reference frame for understanding what a harness does and how components compose. The section on adaptive harness optimization—evolution agents mutating the harness itself based on telemetry—is precisely the meta-harness pattern where the system that wraps the agent is itself modified by an agent, a key frontier for self-improving pipelines. The survey's open problems (regression-free self-evolution, transactional shared state, harness-level evaluation) define the unsolved engineering agenda that any meta-harness project must engage.