Compile, Then Page: Executable SOP Programs and a Capability-Gated Runtime for Procedural LLM Agents¶

🕒 Published (v1): 2026-07-13 10:07 UTC · Source: Arxiv · link

Why this paper was selected

SOPs compiled to executable pseudo-code + capability-gated stack runtime; novel procedural-agent architecture

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper compiles Standard Operating Procedures (SOPs) from natural-language prose into executable two-layer pseudo-code programs, then runs them via a stack-machine runtime that pages only the active frame into the LLM's context. On SOPBench Bank tasks, the pipeline raises pass rate from 70.4% (official text) → 86.4% (compiled text) → 92.8% (compiled + runtime), with 100% refusal correctness, but runtime gains are capability-gated: strong models benefit while weak models lose 14–26 points.

Problem¶

Deploying LLM agents on safety-critical SOPs (banking, healthcare, service desks) is dominated by the practice of prompting the full SOP as resident text (~$10^4$ characters/turn). This strips executable structure—alternative evaluation order, gate short-circuits, and the separation between constraint predicates and their verification recipes—causing skipped checks, premature actions, and incorrect refusals. Even the best official SOPBench configuration reaches only 90.7% refusal accuracy, an error rate unacceptable in compliance contexts.

Method¶

Offline compiler (LLM-free, ~0.1 s for 830 programs). A deterministic compiler translates the benchmark's machine-readable dependency trees into a two-layer pseudo-code program: process functions (one per user goal) call rule subroutines that pair the official constraint wording with a concrete verification recipe and return $r = (h, e)$ (holds + evidence). Five transformations make the output executable: stateful multi-turn subroutines; transitive verifier-gate hoisting via a least-fixed-point $H_T = \text{lfp}_X\,\Gamma(A_R(T)\cup A_G(X))$; evidence-complete scheduling for policy disjunctions; frame factoring of compound groups; and evidence-bearing return contracts. Proposition 1 proves semantic and schedule preservation for faithful executions.

Online program-guided (PG) runtime. A symbolic stack machine executes the compiled program through a single bookkeeping tool (pg_program_step) with operations {start, call, return, select_branch, emit_artifact, complete, fail}. Configuration $\sigma = (S, \Lambda)$ (frame stack + append-only trace); transitions are: $$S\cdot\phi \xrightarrow{\text{call }g(x)\to z} S\cdot\phi\cdot\langle g,0,x,z\rangle, \qquad S\cdot\phi\cdot\langle g,\kappa',\eta',z\rangle \xrightarrow{\text{return }v} S\cdot\text{resume}(\phi,z,v).$$ The runtime injects only $R(\sigma) = \text{render}(\text{top}(S))$, bounding paged context to $B_P = \max_{f\in F(P)}|\text{render}(f)|$ (Bank mean: ~570 chars vs. ~$10^4$ resident). Enforcement is soft (attention, not permissions): deviations execute but are flagged in the audit trail. State persists outside the dialog, enabling re-disclosure after interruption or context compression.

Three-arm experimental design on SOPBench (6 models, 7 domains): offtext (official SOP prose, resident every turn), flat (full compiled program text, resident every turn), pg (compiled program + PG runtime). The offtext→flat delta isolates compilation; flat→pg isolates the runtime. A diagnostic full-cursor arm retains the complete program but places the active frame first, separating frame salience from selective visibility.

Key Contributions¶

An LLM-free SOP-to-program compiler producing call-graph-closed, evidence-disciplined pseudo-code (all 830 SOPBench artifacts compile).
A stack-paged program-guided runtime implementing call-graph-following, stateful, revocable frame disclosure with a well-bracketed audit trail (Proposition 2).
A six-model, seven-domain attribution study with paired McNemar tests, goal-clustered permutations, and re-runs establishing the capability gate empirically (strong: 58:19 and 75:31 pooled discordant pairs, $p<.001$; weak: $-14$ to $-26$ pts, $p<.001$).
Mechanism ablations isolating generic bookkeeping (flat+protocol, n.s. at all tiers), active-frame salience (full-cursor: +2.8 pts, $p=.021$), and selective visibility (pg vs. full-cursor: +1.9 pts, $p=.044$); corroborated by directed reconstruction probes, unprompted post-goal check audits, and attention density measurements.
Two confirmed simulator defects in SOPBench plus a deployment rule: compile first; enable paging only after a model-level discipline check.

Results¶

Bank (DeepSeek-V4-Flash): offtext/flat/pg = 70.4 / 86.4 / 92.8%; refusal accuracy 68.6→93.0→100% (95% lower bound 95.8%).
Compilation gain (flat vs. offtext): +16.0 pts (26:6, $p<.001$); runtime gain (pg vs. flat): +6.4 pts (9:1, $p=.021$).
Compiled representation never significantly hurt any of the six models; gain concentrates where official prose underperforms.
Strong-model runtime benefit across 7 domains: DeepSeek-V4-Flash 58:19 ($p<.001$), Qwen3.7-Plus 75:31 ($p<.001$); effects survive goal-clustered permutation ($p=.004$ each).
Weak models (Qwen2.5-7B, GPT-4o-mini): pg vs. flat = $-14.4$ ($p=.011$) and $-26.4$ ($p<.001$), confirmed in re-runs ($-24.0$ to $-33.6$, all $p<.001$).
Full-cursor ablation (V4, 7-domain, $n=819$): flat/full-cursor/pg = 88.2/91.0/92.9%; separation is refusal-centered (86.3/91.3/93.9%); execution pass rate virtually unchanged.
pg exceeds every recomputed top-3 official SOPBench configuration (best official: 82.4% / 90.7% refusal) on the clean subset.
Domain calls reduced 3.0→2.8 under pg; wall clock 181/105/132 s for offtext/flat/pg.

Limitations¶

Soft enforcement is a capability bet: correctness depends on the LLM faithfully executing the active frame locally; a model that ignores the program degrades rather than gains.
Capability gate is empirically characterized but not mechanistically derived: the boundary is traced to spontaneous state discipline (probe and attention evidence) rather than a predictive theory.
Benchmark scope: evaluation is on SOPBench with a restricted user-turn budget (3 vs. ~10 official); comparability to the official leaderboard is qualified.
Simulator defects: two confirmed SOPBench simulator/state-transition bugs required task exclusions; one goal remains an empirical exclusion without proved impossibility.
Weak-model harm is severe and irreversible: restoring the full program (full-cursor) does not recover weak-model losses, suggesting the paging mechanism itself exceeds low-tier models' usable control capacity.
Single benchmark domain for primary attribution: Bank (125 tasks) is the main attribution domain; cross-domain results use sparser per-domain sets.

Relevance to Agentic AI / LLM Agents¶

This work directly advances the problem of grounding long-horizon LLM agents in formal procedural constraints—a core challenge in deploying agents in regulated, safety-critical settings. The compile-then-page paradigm introduces a hybrid architecture where symbolic control flow (stack machine) handles bookkeeping and the LLM handles semantic execution, a principled alternative to the dominant "SOP-as-text" pattern. The capability-gated finding—that runtime scaffolding helps strong models but harms weak ones—provides an empirically grounded deployment heuristic absent from prior work on agent scaffolding. The mechanism study (reconstruction vs. state discipline vs. attention allocation) establishes a causal chain connecting agent architecture choices to failure modes, offering a reusable diagnostic methodology for researchers designing procedural agents or benchmarking compliance in tool-augmented settings.