3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse¶

🕒 Published (v1): 2026-07-08 23:11 UTC · Source: Arxiv · link

Why this paper was selected

3100-practitioner causal study on AI coding agents and code review; grounded empirical signal

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper synthesizes 3,100 grey-literature documents (from 38,709 collected) using an LLM-assisted grounded-theory pipeline to construct an explanatory causal model of how coding agents reshape code review. The resulting theory has 26 constructs and 67 relationships and argues that code review is the control point through which a coding agent's effect on software quality is decided—and that the team, not the AI, sets whether that effect is positive or negative. A motivating observational study on GitHub confirms surface trends but shows they flip under defensible analysis choices, motivating the need for mechanistic theory.

Problem¶

Repository-mining studies of agent-authored pull requests report contradictory findings (e.g., opposite signs for PR size and merge rates) and cannot explain the mechanisms producing surface-level trends. Without a causal theory, researchers cannot distinguish competing narratives—review becoming a bottleneck, human review becoming unnecessary, or cognitive debt accumulating—nor design studies capable of adjudicating between them.

Method¶

Observational component: Longitudinal re-scrape of the Agents in the Wild GitHub corpus for agent-authored vs. human-authored PR review activity.

Theory-building component (primary): 1. Corpus collection: 38,709 documents from two registers—Reddit threads (31,079, stratified across 43 subreddits, 2020–2026) and web articles (7,630, retrieved via semantic search over 68 paired directional queries). 2. Relevance filtering: LLM judge (Gemini 2.5 Flash, temperature 0) reduces 23,631 candidates to 13,469 (57% retention; Cohen's \(\kappa = 0.75\) vs. stronger model). Coding restricted to 2025–2026 (~9,100 documents). 3. Open coding: Stratified random sample of 3,100 documents coded with Thematic-LM, a multi-agent pipeline using three coder instances with deliberately distinct lenses (neutral, critical, appreciative) to prevent optimism/pessimism bias. An aggregator maintains a versioned codebook via sentence-embedding similarity. Yields 4,838 codes grounded in 109,951 quotes. 4. Theory construction (manual + LLM-assisted): Authors performed axial and selective coding, using LLMs only as a codebook search engine; iterated until the causal graph stabilized at 26 constructs and 67 relationships (64 directed, 3 contested, where the sign depends on a moderator).

Key Contributions¶

Explanatory causal theory of post-AI code review: 26 constructs, 67 directed/contested relationships, making competing mechanisms explicit as falsifiable propositions \(P_1, P_2, \ldots\)
Scalable, open-source LLM-assisted grey-literature theory-building methodology (Thematic-LM pipeline), released as a replication package
Motivating observational study demonstrating that agent-PR review trends are real but directionally unstable under different operationalizations

Results¶

Agent-authored PRs are merged several times faster than human PRs, attract less review discussion (absolute and per line changed), and are reviewed independently at a lower rate: 40.1% reviewed only by the invoking developer vs. 21.5% for human PRs.
No-review rate for merged agent PRs fell from >50% in mid-2025 to ~14% (matching the stable human baseline) by early 2026—the opposite direction from within-reviewer habituation reported by prior work.
Trend direction reverses under equally defensible operationalizations (e.g., whether the invoking developer counts as an independent reviewer).
Causal model organizes constructs into three layers: drivers (e.g., output volume/velocity, surface plausibility, code opacity, data/IP risk), review dynamics (review depth, efficiency, motivation, governance policy, reviewer skill, comprehension debt), and outcomes (throughput, latency, code quality, security, maintainability, knowledge transfer).
Key falsifiable proposition (P3): higher review load increases relative time spent reviewing, which decreases review motivation.

Limitations¶

Corpus may contain LLM-generated documents; the study captures practitioner argument, not verified ground-truth practice.
LLM coding and filtering introduce residual miscoding and hallucination risk despite three independent coders, verbatim quote anchoring, and author auditing.
Axial and selective coding is author judgment; a different team could have drawn different constructs and relationships.
Grey literature over-represents vocal early adopters, vendor advocacy, and post-incident hindsight; breadth (thousands of independent sources) mitigates but does not eliminate this bias.
Observational trends are unstable under analysis choices and cannot distinguish between competing causal stories.

Relevance to Agentic AI / LLM Agents¶

This paper directly characterizes the downstream organizational impact of deploying coding agents at scale—specifically how agentic PR authorship strains or transforms the human oversight layer that follows code generation. The central claim that code review is the control point for agent output quality provides a concrete framework for system designers deciding how much autonomy to grant agents and where to enforce human-in-the-loop checkpoints. The causal model's constructs (surface plausibility masking errors, comprehension debt, reviewer skill erosion) are actionable failure modes for agentic coding systems and directly complement empirical benchmarking of agent code quality. The LLM-assisted theory-building methodology is also a transferable research tool for other domains where practitioner discourse about agentic AI is accumulating faster than it can be analyzed by hand.