From Prompts to Contracts: Harness Engineering for Auditable Enterprise LLM Agents

🕒 Published (v1): 2026-07-09 01:08 UTC · Source: Arxiv · link

Why this paper was selected

Harness engineering for auditable enterprise LLM agents; contracts+source boundaries for production

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper presents a harness-engineering methodology for converting prompt-dominant LLM prototypes into auditable enterprise agents by moving behavioral contracts—source grounding, entity routing, output hygiene, trace generation—out of prompts and into versioned code artifacts. The approach introduces a source-to-claim authority pipeline where only provenance-bound, schema-validated claims may enter runtime answers, enforced by a replaceable composition boundary that wraps any hosted LLM. Empirical evaluation on an investment-briefing agent over 25 Korean listed companies demonstrates that code-owned enforcement preserves both safety and full utility, while prompt-only and bolt-on guardrail alternatives either fail to block violations or over-refuse.

Problem

Production LLM applications commonly begin as "prompt-dominant" prototypes where critical product behaviors—source boundaries, entity routing, answer structure, output hygiene—are embedded in natural-language instructions rather than enforced by code. Such prototypes cannot guarantee reproducible, auditable behavior: prompts carry intent but cannot enforce contracts. Existing frameworks (RAG, guardrails, orchestration runtimes) address parts of this gap but leave source-authority, claim-level provenance, and per-answer contract enforcement to the application layer. The key unmeasured question is whether enterprise contracts are enforced at a composition boundary or merely instructed via prompts.

Method

The harness-engineering approach structures the agent around four explicit layers:

1. Source-to-claim pipeline: Raw documents pass a source gate (manifest registration with URL/filing ID, scope, checksum, runtime policy). Extracted text produces candidate claims that become source-backed claims only upon promotion—atomic, provenance-bound, entity-scoped statements with a runtime-use policy. Only promoted claims may enter an answer; the model cannot draw on unbounded retrieval.

2. Knowledge layer: An LLM Wiki (maintained markdown pages per entity) sits between raw sources and query-time answers as a concise context layer. It is advisory, not authoritative; manifests and source-backed claims remain the runtime source of truth.

3. Replaceable composition boundary: A deterministic schema-checked composer fills answer sections from source-backed claims with no model call (replayable baseline). A live LLM may substitute at this boundary to generate phrasing, but its output must pass the same output-contract gate; if it fails, the harness falls back to the deterministic composer. Three live hosted models were evaluated across 30 scenarios × 3 repeats = 270 runs via OpenRouter.

4. Validation gates: Three check families—leakage (blocks internal identifiers, raw trace records, fixture labels), link (verifies cited sources resolve to public locators), and language (enforces insight-first structure, blocks buy/sell/target-price phrasing)—are implemented in code. A fault-injection control confirms validators flag deliberately broken contracts.

Enforcement-layer ablation: Holding the model fixed, the paper compares (a) full harness, (b) prompt-only enforcement, and (c) bolt-on external guardrail across 120 scenarios.

Key Contributions

• A harness-engineering method for reconstructing prompt-dominant LLM prototypes into traceable, auditable agent architectures with versioned source, control, and validation artifacts.
• A source-to-claim knowledge pipeline that establishes a provenance-bound admission boundary: claim promotion is the point at which the code-owned layer—not the model or retrieval—decides what statements may be asserted as fact.
• A replaceable composition boundary that separates deterministic harness control from LLM phrasing, supporting model substitution without changing contracts.
• A system-level validation design covering source grounding, entity routing, trace completeness, output hygiene, runtime-interface behavior, and latency, plus a live-LLM composition-boundary check and an enforcement-layer ablation with quantitative comparison.

Results

• RQ1 (contract preservation): All harness contracts passed across the fixed validation scenario set; fault-injection control confirmed validators flagged deliberately broken contracts.
• RQ2 (model substitution): Across three hosted models on 270 composition-boundary runs (30 scenarios × 3 repeats × 3 models), harness-side contracts passed on all 270 runs; failures were confined to the model-composed side and were caught and recorded by the harness.
• RQ3 (load-bearing enforcement): Enforcement-layer ablation on 120 scenarios:
• Full harness: 120/120 utility preserved; recommendation-language and internal-trace-leakage violations blocked entirely.
• Prompt-only: violations (recommendation-language, leakage) reached the reader.
• Bolt-on external guardrail: violations blocked but utility dropped to 88/120 due to over-refusal.
• Only code-owned enforcement preserved both full safety and full utility.

Limitations

• Evaluation is conducted on a single, bounded, public-data domain (investment briefing for 25 Korean listed companies, 113 runtime claims), limiting generalizability to other enterprise verticals or data regimes.
• The study evaluates system-level contract verifiability, not investment-decision quality; domain-value validity is explicitly out of scope.
• The validation set is fixed and pre-commercial; no customer interaction logs were available, so adversarial or distribution-shifted inputs are not tested.
• The claim-promotion step is manual or semi-manual; scalability of the source-to-claim pipeline to large, dynamic corpora is not characterized.
• Live-LLM evaluation used OpenRouter as routing infrastructure with a fixed temperature; results may vary under different inference configurations.
• The external guardrail baseline is unspecified by name, limiting reproducibility of the bolt-on comparison.

Relevance to Agentic AI / LLM Agents

This paper directly addresses the productization gap between LLM agent demos and deployable, auditable enterprise systems—a central open problem in agentic AI engineering. The replaceable composition boundary pattern is particularly relevant: it operationalizes the principle that model behavior should be constrained by code-owned contracts rather than emergent prompt compliance, which has direct implications for reliability and safety in tool-calling, multi-step, and multi-agent pipelines. The enforcement-layer ablation provides concrete empirical evidence that prompt instructions and bolt-on guardrails are insufficient substitutes for structural enforcement, strengthening the case for harness-first agent design. For researchers tracking LLM agents, this work extends the harness engineering thread (complementing DSPy, NeMo Guardrails, LangGraph) by contributing a measured, claim-level authority layer and a replicable validation protocol.