From Prompts to Contracts: Harness Engineering for Auditable Enterprise LLM Agents

🕒 Published (v1): 2026-07-09 01:08 UTC · Source: Arxiv · link

Why this paper was selected

Harness engineering framing for enterprise: prompt-to-contract pipeline, auditable agent design

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper formalizes "harness engineering" as a method for converting prompt-dominant LLM prototypes into auditable enterprise agents by relocating product behavior from natural-language instructions into code, manifests, schemas, and validation contracts. The harness wraps a replaceable LLM composition boundary with source-gating, entity routing, claim-eligibility enforcement, and trace generation. Empirically, the authors show that code-owned enforcement is strictly load-bearing: prompt instructions alone cannot reproduce the safety guarantees, and a bolt-on external guardrail achieves safety only by sacrificing utility.

Problem

Enterprise LLM applications begin as "prompt-dominant" prototypes where source policy, entity routing, answer structure, and output hygiene are encoded in natural language. Prompts carry behavior well enough to demonstrate it but cannot guarantee it: they do not enforce source provenance, prevent internal-trace leakage, bind answers to the correct entity, or produce versioned, replayable audit artifacts. Productization requires these properties, and no prior work had measured whether a code-owned harness layer enforces them in ways that prompts alone cannot replicate.

Method

The method introduces a layered architecture with three tiers:

1. Source-to-claim pipeline: Raw documents are registered as source manifests (with URL/filing ID, scope, policy, checksum). Text is extracted into evidence records. Only explicitly promoted source-backed claims—atomic, entity-scoped statements bound to a manifest and evidence record—may enter the runtime answer context. This promotion step is the admission gate; the model never accesses unbounded retrieval.

2. Code-owned control layer: Source eligibility, entity routing (alias/market-ID reconciliation), claim selection, answer structure (insight-first contract), follow-up filtering, trace generation, and three families of validation checks (leakage, link, language) are all implemented in code/schemas, not in the prompt. The prompt is kept short and policy-oriented.

3. Replaceable composition boundary: The LLM is attached only for phrasing. A deterministic, schema-checked template composer provides the fallback baseline; the harness falls back to it if the live-LLM output fails the output contract. The boundary is designed to be runtime-agnostic (AutoGen, LangChain, LangGraph, bespoke loop).

Reference implementation is TypeScript with JSON artifacts covering 5 Korean corporate groups (25 companies, 113 source-backed runtime claims), evaluated across 30 fixed validation scenarios and a live-LLM composition-boundary check with 3 hosted models × 30 scenarios × 3 repeats = 270 runs (via OpenRouter).

Key Contributions

• A harness-engineering method for reconstructing prompt-dominant prototypes into traceable, auditable LLM-agent architectures with versioned control artifacts.
• A source-to-claim knowledge pipeline that defines a strict promotion boundary between raw documents and runtime-eligible, provenance-bound, entity-scoped claims.
• A replaceable composition boundary separating deterministic harness governance from LLM phrasing, with deterministic fallback.
• An enforcement-layer ablation that empirically demonstrates code-owned contracts are load-bearing: prompt-only baselines leak recommendation-language and internal-trace violations; a bolt-on external guardrail blocks violations but over-refuses (utility 88/120 vs. harness 120/120).
• A system-level validation design covering source grounding, entity routing, trace completeness, output hygiene, runtime-interface state, and latency, framed as per-answer contract checks rather than aggregate answer-quality metrics.

Results

• RQ1 (contract preservation): All fixed validation scenarios passed source-grounding, entity-routing, trace-completeness, output-hygiene, and recommendation-language checks; fault-injection controls confirmed validators correctly flagged deliberately broken contracts.
• RQ2 (model substitution): Across 3 hosted models × 30 scenarios × 3 repeats = 270 composition-boundary runs, harness-level checks passed on all 270; failures were on the LLM-composed side and were caught, recorded, and fell back to the deterministic composer.
• RQ3 (load-bearing enforcement ablation):
• Prompt-only baseline: recommendation-language and internal-trace-leakage violations reached the reader.
• Bolt-on external guardrail: blocked violations but over-refused, reducing utility from 120/120 to 88/120.
• Code-owned harness: 120/120 utility with zero contract violations—the only configuration preserving both safety and full utility.

Limitations

• Evaluation is on a single bounded domain (Korean corporate investment briefing, 25 companies, 113 claims); generalizability to other enterprise verticals is not demonstrated.
• The claim-promotion step requires manual or semi-manual annotation; scalability of the admission gate to large corpora is not evaluated.
• Validation is pre-commercial on a research prototype without customer logs; real-world adversarial inputs and distribution shift are not tested.
• Model-substitution check uses hosted models via OpenRouter under fixed temperature; results may not generalize to fine-tuned or instruction-tuned variants.
• The bolt-on guardrail baseline is not identified by vendor/model, limiting reproducibility of the utility-drop comparison.
• Source-backed claims are static at evaluation time; temporal update of the claim set (stale claims, superseded filings) is described architecturally but not measured.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct, foundational contribution to harness engineering as a named discipline: it formally defines the harness as a code-owned control layer that governs a replaceable LLM composition boundary via source manifests, claim eligibility, output contracts, and validation gates, explicitly positioning itself within the emerging harness-engineering literature (citing the unified survey and automation work). For researchers tracking harnesses/meta-harnesses, it provides the first empirical ablation distinguishing code-owned harness enforcement from prompt-based instructions and bolt-on guardrails at the product level, establishing that harness-owned contracts are strictly necessary for simultaneous safety and utility. The architecture's composition-boundary abstraction—designed to be runtime-agnostic across AutoGen, LangChain, LangGraph, and bespoke loops—is directly relevant to meta-harness designs that must coordinate multiple orchestration substrates.