Skip to content

From Human-Centric to Agentic Code Review: The Impact of Different Generations of Generative AI Technology on Review Quality

🕒 Published (v1): 2026-07-14 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

Empirical study of AI-driven vs human code review across gen-AI generations; quality benchmarking

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This large-scale empirical study analyzes 1.02 million pull requests from 207 GitHub projects to characterize how code review quality and efficiency change as teams transition from human-only to LLM-assisted to agentic review. Three distinct AI adoption practices emerge, and while agent-involved collaboration patterns improve review speed, they do not improve—and often worsen—review quality as measured by review smells.

Problem

There is no large-scale empirical evidence on how shifting from human-centric code review to LLM-assisted and agentic code review affects review efficiency and quality. Prior work on human code review factors does not extend to human-AI collaboration dynamics, leaving practitioners without guidance on how to adopt AI reviewers without degrading review quality.

Method

The authors collect all reviewed PRs and review conversations (via GitHub REST API) from 207 open-source projects with continuous review activity from May 2022–February 2026. Each reviewer account is labeled as human, rule-based bot, ML tool, LLM reviewer, or AI agent reviewer through manual inspection of official tool documentation; 384 agent-era PRs are manually validated for agentic behavior (93.75% confirmed). Review eras (pre-LLM, LLM, agent) are defined per-project based on first participation of each reviewer type, retaining only projects with >400 PRs per era. AI adoption practices are identified via soft-DTW time-series clustering on monthly era-specific AI reviewer participation rates \(P_{m,e}\), resampled to 10 evenly-spaced points per era. PR types are classified with GPT-4.1-mini (Cohen's \(\kappa = 0.91\) vs. manual labels). Review quality is measured by: (1) review efficiency = \((T_\text{decision} - T_\text{creation}) / \text{KLOC}\); (2) six review smells adapted from Doğan and Tüzün (Sleeping Review, Review Buddies, Large Changeset, PingPong, Missing Context, Lack of Review). Reviewer interaction sequences within each PR are mined to identify human-AI collaboration patterns. Explanatory logistic regression models (RQ3) jointly assess human-AI collaboration patterns alongside traditional factors (PR characteristics, review activity, participant experience by reviewer type).

Key Contributions

  • Large-scale longitudinal dataset of 1.02 million PRs across three generative AI review eras; replication package released.
  • Identification of three empirical AI adoption practices via time-series clustering: Gradual AI Adoption (46% of projects), Rapid LLM Adoption (22%), Rapid AI Agent Adoption (32%).
  • Quantification of how adoption practice, not merely AI participation, determines review efficiency and smell outcomes.
  • First large-scale quantitative study of agentic code review, modeling reviewer interaction sequences to characterize human-AI collaboration patterns.
  • Demonstration that human-AI collaboration patterns become the strongest explanatory factor for review efficiency once generative AI reviewers are present.

Results

  • Gradual AI Adoption: Review time drops 2.5 days/KLOC (pre-LLM→agent era, significant); no significant change in review smell rate.
  • Rapid AI Agent Adoption: Review time drops 4.5 days/KLOC (significant); Review Buddies smell rises +16.6 pp in LLM era, +12.4 pp in agent era (significant).
  • Rapid LLM Adoption: No significant change in review efficiency across eras; review smell rate rises +8.0 pp in LLM era and +4.4 pp in agent era (significant); Review Buddies rises +26.0 pp (LLM era) and +23.2 pp (agent era)—the sharpest quality degradation across all practices.
  • Agent-initiated or multi-agent collaboration patterns are significantly more efficient than human-only review under Gradual AI Adoption and Rapid AI Agent Adoption, but most LLM/agent collaboration patterns show significantly higher review quality risk than human-only review.
  • No human-AI collaboration pattern consistently outperforms human-only review on both efficiency and quality simultaneously.
  • In RQ3 explanatory models, human-AI collaboration patterns emerge as the strongest factor for the Review Buddies smell once generative AI reviewers join; agent-involved patterns are disproportionately used on larger changesets.

Limitations

  • Study is observational; causal claims about AI adoption causing quality changes cannot be made.
  • Projects must have >400 PRs per era, biasing the sample toward larger, more active repositories; small projects or those with short transition windows are excluded.
  • Review smells are proxies for quality risk, not direct measures of defect escape rates or downstream bug outcomes.
  • Reviewer labeling relies on official tool documentation; tools that combine LLM and agent capabilities may be miscategorized.
  • 24 of 384 sampled agent-era PRs showed no agentic behavior evidence beyond natural-language comments, introducing minor label noise.
  • Results may not generalize beyond open-source GitHub projects.
  • The text is truncated before full RQ2/RQ3 results and the complete smell analysis are reported.

Relevance to Agentic AI / LLM Agents

This paper provides the first large-scale empirical benchmark of agentic AI behavior in a real-world software engineering workflow, directly measuring how AI agents (e.g., Claude Code, Copilot) interact with human developers in code review at scale. The finding that efficiency gains from agent-involved review do not translate into quality improvements—and that rapid LLM adoption degrades review diversity (Review Buddies)—has direct implications for how agentic systems should be integrated into collaborative human-AI workflows. The modeling of reviewer interaction sequences as collaboration patterns offers a framework applicable to other agentic task domains beyond code review. For those tracking agentic AI, this study establishes an empirical baseline for measuring agent-era effects and highlights the unsolved tension between agent throughput and review quality assurance.