Harness as an Asset: Enforcing Determinism via the Convergent AI Agent Framework (CAAF)

🕒 Published (v1): 2026-04-18 15:15 UTC · Source: Arxiv · link

Why this paper was selected

Determinism enforcement in harnesses via CAAF; safety-critical deployment

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CAAF (Convergent AI Agent Framework) enforces deterministic constraint compliance in LLM-based engineering workflows by treating a versioned domain-invariant registry—the "Harness"—as a first-class enterprise asset backed by a deterministic Unified Assertion Interface (UAI). Three interdependent pillars (Recursive Atomic Decomposition, Harness-as-an-Asset, Structured Semantic Gradients with State Locking) close what the authors call the "controllability gap" that sycophancy, context rot, and stochastic oscillation create in open-loop LLM orchestration. Evaluated on autonomous-driving and pharmaceutical-reactor benchmarks, CAAF achieves reliable constraint satisfaction on commodity-tier models where monolithic GPT-4o collapses to 0% accuracy without a prompt hint.

Problem

LLMs deployed in safety-critical engineering (automotive, biopharma, cloud infra) suffer three structural failure modes: (1) Compliant Hallucination — outputs that are syntactically persuasive but physically contradictory, driven by sycophancy and completion bias; (2) Context Rot — softmax attention dilutes early-defined safety constraints as context grows, empirically confirmed even in Claude Opus 4 with high-effort thinking (20/20 trials fail a forward-safety check under cross-domain noise); (3) Stochastic Oscillation — unstructured self-reflection fixes one constraint while re-violating another, producing an unbounded random walk. Existing techniques (CoT, ReAct, agent debate) do not architecturally address any of these, making reliability a prompt-engineering artifact rather than a system property.

Method

CAAF adds three pillars on top of commodity LLM inference, touching no model weights:

Pillar 1 — Recursive Atomic Decomposition (RAD) + Context Firewall. The Orchestrator has an LLM decompose requirements into a JSON node graph with explicit parent_id dependency links; a deterministic topological sort then sequences execution. Each Executor runs in a physically isolated API thread with only the harness slice and context keys declared in its scope — preventing cross-domain contamination (e.g., a cost node cannot read a safety node's context). A Progressive Graph Construction mode spawns sub-orchestrators for large specifications.

Pillar 2 — Harness as an Asset (HaaA). Domain invariants are extracted from human experts and regulatory frameworks into versioned YAML constraint files bound to a Unified Assertion Interface (UAI) — a Python eval() engine that returns PASS or FAIL + Error Trace deterministically. The UAI also integrates external validators (simulation, EDA, HiL, TLA+). The harness lifecycle includes: tacit-to-explicit transformation, meta-validation via CI/CD with golden/poisoned test artifacts, freezing under RBAC, and a Change Approval Board (CAB) for any relaxation. The Harness is explicitly framed as a compounding corporate asset whose value grows as inference commoditizes.

Pillar 3 — Structured Semantic Gradients + State Locking. When a UAI check fails, a Semantic Reviewer computes a gradient triple: $\(\vec{\nabla}\epsilon = \{\text{Dimension}_i,\ \text{Direction}_i,\ \text{Exact Boundary Magnitude}_i\}\)$ The Dimension and Direction are anchored deterministically by the UAI boolean failure and error trace; the Magnitude is LLM-inferred but immediately re-checked by the UAI if incorrect (closed-loop correction). State Locking marks every PASS-ed JSON schema node read_only: true in subsequent iteration prompts, enforcing monotonic growth of the verified constraint set \(V_t \subseteq C\) across iterations and preventing oscillation. Feedback operates at two levels: per-node (Executor retry) and global (cross-node integration after all nodes converge).

Key Contributions

• Harness-as-an-Asset thesis: formalizing domain invariants into a versioned, auditable YAML+UAI registry that outlives any specific model generation and enables fully on-premises deployment for regulated sectors (ISO 26262, ISO 13485, IEC 62443, SOTIF).
• UAI ablation demonstrating that deterministic grounding alone is necessary but insufficient at commodity model cost — oscillation persists until State Locking is added; frontier reasoning can substitute for Pillar 3 but at ~10× higher per-trial cost.
• Monotonic non-regression proof sketch: State Locking guarantees \(V_t \subseteq V_{t+1}\), giving finite termination or paradox detection rather than oscillation.
• Cross-domain replication: results on closed-source GPT-4o replicated on two independent open-weight families across automotive (2-constraint L3 paradox) and pharmaceutical (7-constraint flow-reactor, 3-way minimal unsatisfiable subset) benchmarks.
• Strategic Negotiation / Paradox Detection: when constraints are irreconcilable, CAAF terminates with a deadlock report and a UAISolver-anchored menu of human-readable trade-off options rather than silently relaxing constraints.

Results

• Monolithic GPT-4o with explicit "check for paradox" hint: 90% accuracy on controlled L3 AD engineering task; without the hint (deployment-realistic): 0% accuracy.
• CAAF on commodity-tier model: achieves reliable constraint satisfaction on both benchmarks (exact pass rates per tier reported in Sections 4–9 of the full paper; summary in ablation — deterministic UAI alone still collapses into oscillation at commodity cost; UAI + State Locking closes the gap without frontier models).
• Claude Opus 4 with adaptive high-effort thinking (n=20) commits the forward-safety violation in 20/20 trials with zero variance under cross-domain noise, despite mean encrypted-thinking signatures >6,000 characters.
• Multi-agent baselines (debate, sequential checking) fail across all domains tested without UAI grounding.
• Frontier reasoning (e.g., o-series) can substitute for Pillar 3 but at ~10× per-trial cost versus commodity + State Locking.

Limitations

• Only two benchmark domains reported (automotive AD paradox, pharmaceutical flow reactor); broader empirical validation is explicitly deferred as future work.
• Harness fidelity is the determinism ceiling: a flawed constraint causes deterministic convergence toward the wrong answer; no automated harness correctness guarantee is provided beyond CI/CD meta-validation.
• Magnitude component of the Structured Semantic Gradient remains LLM-inferred and stochastic; convergence speed depends on LLM quality even if direction is anchored.
• Harness construction requires significant upfront expert labor (tacit-to-explicit transformation) and a CAB governance process, creating organizational overhead.
• Progressive Graph Construction for large specifications is described architecturally but not empirically evaluated.
• No new training, architecture, or scaling contribution — the framework is entirely at the orchestration layer and inherits all inference-layer limitations.

Relevance to Harnesses / Meta-Harnesses

CAAF is directly on-topic: it elevates the harness from a test scaffold to a versioned enterprise asset with its own lifecycle (meta-validation, RBAC freezing, CAB change control), making it arguably the most explicit articulation in the literature of a harness-centric architecture for AI workflows. The "Harness-as-an-Asset" framing — where domain invariants in a YAML+UAI registry compound in value as the inference layer commoditizes — maps cleanly onto meta-harness thinking: the orchestration logic is domain-agnostic and reusable across domains, while only the harness registry is domain-specific. The three-pillar ablation is particularly useful for harness researchers because it empirically isolates the contribution of deterministic grounding (UAI) from convergence control (State Locking), showing that a harness alone without a convergence mechanism is insufficient. The industrialization thesis — self-hosted, on-premises deployment enabled by harness-anchored reliability on commodity models — is a concrete argument for why harness engineering infrastructure matters independently of model capability trends.