Skip to content

AgentSPEX: An Agent SPecification and EXecution Language

🕒 Published (v1): 2026-04-14 23:16 UTC · Source: Arxiv · link

Why this paper was selected

Agent specification and execution language; foundational DSL for harness programming

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AgentSPEX is a YAML-based declarative language for specifying LLM-agent workflows with explicit control flow, modular composition, and a customizable execution harness. It decouples workflow logic from Python, addressing the coupling problem in frameworks like LangGraph and DSPy while outperforming ReAct baselines on 7 benchmarks across science, math, writing, and software engineering domains.

Problem

Reactive (ReAct-style) agents rely on a single system prompt to govern open-ended tool-use sequences, leaving control flow and state implicit — leading to poor reproducibility, high cost, and degraded performance on long-horizon tasks. Existing structured frameworks (LangGraph, DSPy, CrewAI) impose explicit workflows but tightly couple them to Python, creating steep learning curves and limiting shareability with non-programmers. Neither paradigm provides explicit context management to combat "context rot" as conversation history grows.

Method

AgentSPEX defines a lightweight YAML vocabulary (Table 1) covering invocation (task, step), control flow (if/switch, while, for_each), composition (call), concurrency (parallel/gather), and state (set_variable, return). Workflows use Mustache-style templates ({{variable}}) for explicit context passing between steps. The agent harness executes these specifications via: - An interpreter that validates structure, resolves templates, expands nested constructs with hierarchical step IDs (e.g., 3.2.1), and dispatches to typed step handlers. - An executor running a multi-turn loop per task/step: send message history → call LLM → execute tool calls via MCP client → append results → repeat until no tool calls or token/call limit. - A Docker sandbox with browser, filesystem, code execution, and 50+ MCP tools. - Checkpointing after each step (saves context variables, sandbox state, metrics) with resume from any checkpoint. - Selective trace replay: load \(N\) prior steps from a saved trace and resume live execution, isolating the effect of prompt changes on downstream behavior without re-running upstream steps. - A bidirectional visual editor with synchronized YAML and interactive flowchart views.

The task construct starts a fresh conversation (no history carry-over); step accumulates turns. This task/step distinction gives authors direct control over information flow — a key mechanism for managing context size.

Key Contributions

  • YAML-native agent specification language with typed constructs for full workflow control (branching, loops, parallelism, composition), executable without modifying Python source.
  • Unified submodule abstraction: skills and agents are both YAML workflows, composable via call; workflows can also be registered as LLM-callable tools at runtime.
  • Explicit per-step context management (Mustache templates + save_as) rather than implicit growing conversation history.
  • Agent harness with Docker sandbox, MCP tool integration, observability dashboard, checkpointing, and selective trace replay for development iteration.
  • Formal verification pathway: explicit step boundaries and variable dependencies enable pre/post-condition specification in Lean or Isabelle.
  • Bidirectional visual editor with drag-and-drop node editing synchronized to YAML source.
  • Three production-ready agents (Deep Research, AI Scientist, AI Advisor) distributed as reference implementations.

Results

All results from Table 2; AgentSPEX tops all 7 benchmarks:

  • SciBench (chemistry, GPT-5): AgentSPEX 90.61% vs. ReAct 87.79% vs. CoT 85.92% (+2.82pp over stronger baseline)
  • StemEZ (physical chemistry, GPT-5): 86.57% vs. ReAct 84.72% (+1.85pp)
  • ChemBench (GPT-5, high-reasoning): 83.30% vs. CoT 78.90% (+5.5pp; ReAct 77.80% underperforms CoT)
  • AIME 2025 (GPT-5): 100.0% vs. CoT+Python 99.60% vs. CoT 94.60%
  • ELAIPBench (paper understanding, GPT-5): 43.70% vs. CoT 37.22% (+6.48pp; ReAct 33.80% underperforms CoT by 3.4pp)
  • WritingBench (Claude-Sonnet-4.5-Thinking): 81.00% vs. ReAct 80.30% vs. CoT 79.90%
  • SWE-Bench Verified (avg. Claude-Opus-4.5/4.6, high-reasoning): 77.1% vs. mini-SWE-agent 76.2% vs. Live-SWE-agent 74.60%
  • User study (n=23): AgentSPEX rated more interpretable and accessible than LangGraph; LangGraph preferred for complex workflows.

Limitations

  • Workflows are currently hand-authored (with coding assistant aid); no automated workflow synthesis from task description.
  • User study participants (n=23) expressed less confidence in AgentSPEX for highly complex workflows, favoring LangGraph for intricate multi-step scenarios.
  • Formal verification support is described as a future/potential direction, not a shipped feature.
  • Multi-agent orchestration is limited; parallel subagent coordination and cross-agent communication are identified as future work.
  • Context compression for long-horizon tasks is not yet implemented within the harness.
  • No end-to-end training pipeline to train models on AgentSPEX-formatted workflows.

Relevance to Harnesses / Meta-Harnesses

AgentSPEX is a direct instantiation of the harness design pattern: a structured execution environment that wraps LLM calls with sandboxing, checkpointing, selective replay, logging, and tool dispatch, entirely separated from the workflow specification layer. The task/step distinction and explicit context variable system are a concrete mechanism for harness-level context management — a central unsolved problem in meta-harness design. The selective trace replay feature is particularly relevant to harness engineering workflows, enabling iterative prompt refinement without full re-execution. The paper also benchmarks the harness value proposition directly: the ReAct baseline uses the same workflow spec but without harness enforcement, and consistently underperforms — isolating the harness's contribution to structured execution.