Skip to content

Copy-on-Write Scoring: Application-Specific Agent Evaluations

🕒 Published (v1): 2026-07-15 19:59 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML 2026; copy-on-write evaluation granularity for application-specific agent workflows

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Copy-on-Write (CoW) Scoring is an evaluation framework that lets LLM-based agents execute directly inside a live PostgreSQL application by intercepting all writes at the database-view level, isolating them per session without touching production data. Scores are produced at both session level (did the agent reach the correct final world state?) and operation level (how much did each operation contribute?). The framework closes the construct-validity gap of external benchmarks by making application-specific evaluation cheap enough to iterate on.

Problem

Existing agent evaluation methods have low construct validity for specific deployed applications: generic benchmarks do not predict behavior in real systems, and replica/simulation environments are expensive to build, drift from production state, and still do not transfer across applications with different tooling and system prompts. Teams need a way to evaluate agent database writes directly in their own application environment without risking production data corruption and without the overhead of maintaining a separate replica.

Method

CoW Scoring rewrites the database schema transparently: each original table \(T\) is split into a base table \(T_{\text{base}}\) (immutable production data), a changes table \(T_{\text{changes}}\) (per-session agent writes, augmented with session_id, operation_id, cow_updated_at, cow_deleted), and a SQL view \(T\) defined as base ∪ changes filtered by session. INSTEAD OF triggers on the view redirect all agent writes to \(T_{\text{changes}}\), leaving \(T_{\text{base}}\) untouched. Evaluation compares a human-recorded ground-truth (GT) session against an agent session along two dimensions and two granularities:

  • Session-level structural score: \(s_{\text{struct}} = \frac{N_{\text{matched}}}{N_{\text{matched}} + N_{\text{missing}} + N_{\text{extra}}}\)
  • Session-level content score: \(s_{\text{content}} = \frac{1}{N_{\text{matched}}} \sum_{(g,a) \in \text{matched}} \text{sim}(g, a)\)
  • Op-level structural utility: \(u^{\text{struct}}(o_i) = s_{\text{struct}}(G_{\leq i}) - s_{\text{struct}}(G_{< i})\), computed in topological operation order
  • Op-level content utility: \(u^{\text{content}}(o_i) = \frac{1}{|M_i|} \sum_{(g,a) \in M_i} \text{sim}(g, a)\)

Row matching uses greedy best-match with a UUID mapping to bridge independently generated primary keys. Field similarity defaults are SQL-type-aware (SequenceMatcher for text, deep-equal for JSON, exact for other types); custom comparators are injectable. Session-level scores are path-invariant (only final database state matters).

Key Contributions

  • CoW database mechanism: PostgreSQL view + trigger architecture that isolates agent writes per session with no production-data risk; implemented in the open-source agent-cow library.
  • Two-dimensional, two-granularity scoring: structural and content scores at both session and operation level, enabling localization of exactly where and how an agent diverges from ground truth.
  • Closed evaluation loop on Plane: demonstrated score-diagnose-fix cycle on a real open-source project-management platform across 5 models × 20 workflows × 3 trial rounds (300 sessions total).
  • Operation-level credit assignment: partial-progress scoring suited for use as dense reward signals in RL fine-tuning (GRPO-style) or prompt-optimization loops (GEPA).
  • Low integration cost: ~250 LOC for CoW setup in Plane; ground-truth sessions generated at scale using an LLM (Claude Opus 4.6) plus human validation.

Results

  • Five models evaluated: GPT-5, GPT-4.1, Gemini-3.1-Pro, Gemini-3.1-Flash-Lite, Gemini-2.5-Pro; 200 sessions in runs 1–2, 100 additional sessions in run 3 after tool-surface fix.
  • Within-family ranking matches public benchmarks (Gemini-3.1-Pro > Flash-Lite; GPT-5 > GPT-4.1), providing a sanity check on scoring validity.
  • Per-model mean overall scores are stable across two trials (max delta ≤ 0.03), indicating low run-to-run variance.
  • After fixing a vocabulary mismatch ("work item" vs. "issue") in the tool surface:
  • GPT-4.1: \(+0.47\) (\(0.32 \to 0.79\))
  • Gemini-2.5-Pro: \(+0.54\) (\(0.43 \to 0.98\))
  • Gemini-3.1-Flash-Lite: \(+0.03\) (\(0.80 \to 0.83\)); zero failed sessions in run 3 vs. at least one per earlier iteration
  • GPT-5 and Gemini-3.1-Pro already near ceiling (\(\approx 0.95\)–\(0.96\)), improved by \(\leq 0.02\)

Limitations

  • GT session creation is manual, bounding scalability; automated MCTS-style discovery is proposed but not yet implemented.
  • Scoring assumes fully specified prompts; underspecified real-user prompts with multiple valid final states are conflated with failure.
  • Only write operations are scored; read-operation patterns (e.g., redundant discover calls) carry diagnostic signal that is not captured by current structural/content scores.
  • Mechanism currently assumes PostgreSQL; portability to other data stores (NoSQL, file systems) requires additional engineering.
  • Cascade triggers and PostgreSQL-specific trigger configurations are not yet handled, limiting real-world portability.
  • Single-application demonstration (Plane); generalization claims require evaluation across more applications.

Relevance to Harnesses / Meta-Harnesses

CoW Scoring is directly a testing harness concern: the paper's Plane study builds an explicit agent harness (harness execute and harness discover tools wrapping an OpenAPI-spec vector store), and the CoW mechanism acts as the harness's isolation and observability layer. The score-diagnose-fix loop is precisely a meta-harness workflow — iterating on tool surfaces, system prompts, and models by treating operation-level scores as structured feedback, which the authors explicitly extend toward GEPA-style prompt optimization and GRPO-style RL reward signals. For researchers tracking harness design, this paper provides both a concrete architectural pattern (DB-level CoW as safe write isolation) and a quantitative methodology for evaluating whether a harness's tool surface is adequate, making it directly applicable to any agentic pipeline that writes to a database.