Skip to content

Understanding Agent-Reactive Bugs at the Model-Harness Boundary: An Empirical Study of LLM Agent Issue Reports

🕒 Published (v1): 2026-07-17 06:53 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper presents the first empirical study of agent-reactive (AR) bugs—failures that arise only when a specific LLM-generated response meets a harness that mishandles it. Through manual annotation of 255 AR bug reports across Codex, Gemini-CLI, LangChain, and CrewAI, the authors build a two-axis taxonomy of symptoms and triggering behaviors, then document a systematic mismatch between how users and developers assign fault.

Problem

Prior empirical studies of agent bugs attribute failures either to limited LLM capability (e.g., hallucination) or to harness-side defects (e.g., outdated APIs, configuration misalignment), leaving uncharacterized a third class: bugs that manifest only when the backend LLM emits a particular response and the harness reacts to it incorrectly. These AR bugs cannot be diagnosed by inspecting either component in isolation, and their stochastic triggering condition makes reproduction and fault localization exceptionally difficult.

Method

The authors mine GitHub issue trackers for four open-source agent projects (Codex, Gemini-CLI, LangChain, CrewAI), applying a three-stage pipeline: (1) keyword-based filtering to isolate bug reports whose reproduction steps reach an LLM call (11,288 issues), (2) engagement filtering to retain issues with sufficient community discussion (3,037 issues), and (3) manual dual-annotation by two authors using an open-coding protocol to label each issue as AR or non-AR and, for AR issues, to assign a triggering behavior and symptom category. This yields 255 confirmed AR bugs. Annotators each spent ~150 hours; conflicts were resolved by consensus. The resulting two-axis taxonomy (triggering behavior × symptom) is used to analyze detection difficulty, reproducibility, and fix attribution (RQ1–RQ3).

Key Contributions

  • First empirical study dedicated to AR bugs, defining the construct and separating it from capability failures and pure harness defects.
  • Annotated dataset of 255 AR bugs from two CLI products and two agent frameworks, with full issue-discussion traceability.
  • Two-axis taxonomy pairing five symptom categories with eight LLM triggering behaviors, with per-project distribution analysis (Table II).
  • Characterization of detection, reproduction, and localization challenges specific to AR bugs, motivating test oracle design, auto-reproduction tools, and fault-attribution guidelines.
  • Empirical evidence of a user–developer mismatch: users propose harness-side guardrails, developers often attribute the bug to the LLM or respond slowly to user-proposed fixes.

Results

  • AR bugs constitute 8.4% of actively discussed issues across the four projects.
  • 255 AR bugs identified from a starting corpus of 32,373 issues (3,037 engagement-filtered).
  • Symptom distribution:
  • Silent Errors (SE): 108 (42%) — largest category; require inspecting tool logs or workspace diffs to detect.
  • Crash (Cr): 71 (28%) — 79% of LangChain AR issues; often caused by harness parse failures on unexpected LLM output.
  • Error in Output (EIO): 34 (13%) — concentrated in Gemini-CLI (14) and CrewAI (12).
  • Retry Loop (RL): 33 (13%) — concentrated in CLI products (Gemini-CLI 18, Codex 11).
  • Hang (H): 9 (4%) — all from CLI products; caused by child processes holding open pipes after timeouts.
  • Triggering behavior breakdown (partial, from Table II): Unexpected Tool Arguments (UTA) accounts for 62 bugs across all projects, with no dominant downstream symptom—crash, retry loop, and EIO each take a substantial fraction; Incorrect behavior (INC) is the most frequent trigger in Codex (36 issues).
  • LangChain and CrewAI suffer proportionally more Crash-type AR bugs than CLI products (79% and 25% vs. 26% and 2%).

Limitations

  • Coverage limited to four projects; findings may not generalize to other agent architectures (e.g., multi-agent pipelines, browser-use agents, voice agents).
  • AR bug identification required subjective human judgment; despite dual-annotation and consensus resolution, some borderline cases remain ambiguous.
  • The paper text is truncated before the full RQ2 triggering-behavior analysis and all of RQ3 (fix attribution), so quantitative fix-location findings are not fully reported here.
  • Because LLM behavior is stochastic and version-dependent, the 255 issues may undercount AR bugs that were closed as "cannot reproduce" before annotation.
  • The engagement-filtering threshold was project-specific and heuristic, potentially biasing the sample toward high-visibility bugs.

Relevance to Harnesses / Meta-Harnesses

This paper directly maps the failure modes of the harness layer—prompt rendering, output parsing, tool dispatch, context compaction, and loop control—onto a concrete bug taxonomy, making it essential reading for harness and meta-harness designers. The taxonomy's triggering-behavior axis (e.g., unexpected tool arguments, instruction-ignoring, context-overflow-induced state loss) identifies exactly the harness–model interface points where robustness mechanisms (output validators, retry guards, compaction-aware state recovery) are most needed. The user–developer fault-attribution mismatch implies that meta-harnesses that add observability (tool-call logs, trajectory diffing, oracle hooks) could resolve disputes that currently stall fixes. The finding that silent errors are the dominant symptom—with no exception or warning, requiring external evidence to detect—directly motivates the design of harness-level monitoring and automated test oracles for agent loops.