How Agent Skills Fail under Long Contexts: A White-Box Study in Code Auditing

🕒 Published (v1): 2026-07-20 13:34 UTC · Source: Arxiv · link

Why this paper was selected

White-box study of skill/instruction degradation in long-context agent trajectories; directly actionable for harness design

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This white-box case study examines how LLM coding agents fail to honor mandatory skill instructions when the working context grows long. Using a production-derived code-audit workflow with 24 deterministic checks, it finds that adding ~299K characters of context—regardless of semantic relevance—reduces pass rate from 8/10 to 3/10 on the primary model–task pair. A concrete external checklist that restates all 24 checks recovers full pass rate (10/10) vs. 5/10 for a generic self-validation prompt.

Problem

Reusable agent skills package procedural instructions for coding agents, but loading a skill does not guarantee every stated requirement remains active across a long, tool-heavy trajectory. Prior work shows usable context is smaller than advertised windows, and trajectory studies reveal that failures begin early and remain hidden; yet no focused study has held a fixed skill-based task constant while varying only the surrounding context and then classified exactly which requirement failed and where in the visible trace. This gap is the paper's direct target.

Method

The study adopts a white-box pipeline (Figure 1): a production code-audit skill (operation-sl017 Stage 4CD) with four structured tasks is translated into 24 deterministic predicate checks over saved artifact state. A sanitized, answer-free workspace is rebuilt from scratch for each run. Three main context conditions are compared: - Clean (10,991 chars): minimum complete task context - Relevant long (299,140 chars): same-workflow production material appended - Irrelevant long (299,140 chars): unrelated natural-language archive, character-matched

The primary model is Codex with gpt-5.4-mini at medium reasoning effort; five additional models appear in exploratory probes. Two outcome measures are: $\(\text{RequirementCoverage} = \frac{1}{K}\sum_{i=1}^{K} z_i, \quad \text{TaskSuccess} = \mathbf{I}\!\left[\sum_{i=1}^{K} z_i = K\right]\)$ Failure classification assigns each failed run to the earliest visible evidence class: Lost Requirement, Editing Drift, Failed Checking, or Non-Agent Failure. A mitigation comparison swaps only the completion instruction between a Generic prompt ("validate every constraint") and a Detailed prompt (explicit list of all 24 checks). A scaffold probe varies context delivery (Direct, Bundled-Direct, Full-Direct, Codex with tools) to test whether tool-mediated selective retrieval reduces failures.

Key Contributions

• A controlled white-box study design: task, starting artifact, and 24 checks are fixed; only surrounding context varies.
• A four-class taxonomy of visible failures (lost requirements, editing drift, failed checking, non-agent failures) with explicit decision rules and a reusable classification guide.
• Empirical evidence of a 50-percentage-point observed pass-rate drop under long context on one model–task pair, with counterevidence from a second task that passes all conditions.
• A mitigation result: a detailed external checklist (10/10 pass) significantly outperforms a generic self-check (5/10; two-sided Fisher \(p = 0.0325\)).
• Infrastructure reliability exposed as a separate measurement problem: only 29/57 attempts in the extension produced scoreable outputs.

Results

• RQ1 (long context vs. clean): Clean passes 8/10 (98.8% check coverage); Relevant long passes 3/10 (92.1%); Irrelevant long passes 3/10 (93.8%). Observed failure-rate rise: +50 pp. Both Fisher tests yield \(p = 0.0698\), below the \(\alpha = 0.05\) threshold; Wilson intervals overlap widely.
• RQ2 (relevance): Relevant and Irrelevant long contexts are statistically indistinguishable (Fisher \(p = 1.0\), both 3/10). Semantic domain of filler text does not reliably order failure severity.
• RQ3 (failure taxonomy): Failures are often narrow—losing one required array converts an otherwise 23/24-correct artifact into a failed run. One Irrelevant run completed only 2 of 4 tasks (14/24 checks). Silent failure (agent reports success despite artifact failure) is common.
• RQ4 (checklist intervention): Detailed checklist: 10/10; Generic self-check: 5/10; \(p = 0.0325\). Prompt length difference is only 649 characters; informational difference (specific enumerated checks) is the operative variable.
• RQ5 (scaffold): Exploratory only; coding agents with tool access can reduce effective working set but do not eliminate failures.
• Historical mixed-condition inventory: 97.98% individual-check pass rate despite 44 failed trajectories, confirming sparse failure structure.
• Equivalent cost of the full study: USD 56.02 across 167M logical tokens (lower bound).

Limitations

• Only 10 valid runs per primary condition; Fisher tests trend-level only (\(p \approx 0.07\)); Wilson intervals are wide.
• Extending Irrelevant from 5 to 10 runs shifted the estimate from 0/5 to 3/10, illustrating instability of small-\(n\) agent samples.
• Single primary model–task pair (Codex/gpt-5.4-mini + Stage 4CD); the second task passes all conditions, so no universal length threshold can be claimed.
• Decision to extend was post-inspection, so this is not a preregistered confirmatory experiment.
• Scaffold probe is diagnostic only: different token budgets, retrospective bundle selection, and gateway limitations prevent fair comparison.
• Infrastructure censoring (50.9% attempt-to-scoreable-output rate) is heavily skewed toward the Irrelevant-long condition and may be non-random, confounding operational success estimates.
• Context conditions differ in character count (reproducible) but not necessarily in token count, structural complexity, or retrieval path composition.

Relevance to Harnesses / Meta-Harnesses

This paper directly diagnoses a failure mode of skill-based harnesses: the wrapper that packages instructions does not guarantee their persistence across a long multi-tool trajectory. For meta-harness designers, the central finding is that a detailed external checklist injected at completion time rescues runs that a generic self-check cannot, because a self-generated validator inherits the same active-requirement dropout that caused the original omission. The four-class taxonomy (lost requirements, editing drift, failed checking, non-agent failures) provides a diagnostic vocabulary for instrumenting harness telemetry and post-hoc artifact audits. Infrastructure reliability (only ~51% scoreable attempt rate) is called out as a distinct measurement concern that any harness evaluation framework must account for separately from model capability.