Skip to content

Govern the Repository, Not the Agent: Measuring Ecosystem-Level Risk in AI-Native Software

🕒 Published (v1): 2026-06-26 16:22 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Integration friction in agent-authored code is largely a property of the repository ecosystem, not individual agents. Using multilevel models on ~930K agent-authored PRs, ~50% of friction variation sits at the repository level and survives full controls, with agent-authored contributions concentrating this friction ~2x more than human ones (ICC 0.30 vs 0.16).

Problem

Current agent evaluation (e.g., SWE-bench) assumes compositional correctness — if each agent passes its tests, the system is sound. Practitioners report the opposite: code arrives faster than teams can absorb, shared mental models erode, rationale goes unwritten. The question is whether this belongs to individual agents or to the ecosystem of agents, humans, and automation.

Method

  • Uses multilevel (hierarchical) models to partition variance of integration friction across two levels: pull-request (level 1) nested inside repository (level 2).
  • Integration friction measured via 7 constructs across 4 facets: timing (deliberation latency, resolution latency), effort (review rounds, comment volume), contention (changes requested, conflict incidence via AgenticFlict), outcome (rejection).
  • Non-reducibility metric: intraclass correlation \(ICC = \tau^2 / (\tau^2 + \sigma^2)\), where \(\tau^2\) is repository-level variance and \(\sigma^2\) is within-repository variance. For binary outcomes, \(ICC_{latent} = \tau^2 / (\tau^2 + \pi^2/3)\).
  • Fits 4 nested models (M0–M3) adding PR-level covariates (size, task type, timing, agent indicator), repository covariates (stars, forks, language, activity), then codebase size and age.
  • Primary dataset: AIDev curated subset (33,596 PRs, 2,807 repos ≥100 stars); full set (930,292 PRs, 116,211 repos) for cross-check. Human-authored PRs for the same repos fetched from GitHub API for matched baseline.
  • Tests 4 hypotheses: coupling (H1), locus of friction (H2), non-reducibility (H3), specificity to agent authorship (H4).

Key Contributions

  • Recasts practitioner-reported erosion of shared understanding as a measurement question: at what level should risk be measured?
  • Operationalizes emergence as statistical non-reducibility via ICC, computed with multilevel models the field already uses, plus a matched human baseline showing the signal is specific to agent authorship.
  • Produces a measurement and governance agenda: individual agent capability is necessary but not sufficient for a dependable repository.

Results

  • ~50% of integration friction variation sits at the repository level and survives full controls across multiple constructs.
  • Agent-authored contributions concentrate friction ~2x more than human-authored ones in the same repositories: ICC 0.30 vs 0.16; gap persists after controlling for codebase size, age, task shape, process maturity, and merge path.
  • Coupling is pervasive: 28–34% of engaged PRs draw reviews/comments from human accounts; 46–53% from automated bots.
  • Friction is driven by base-branch churn, not number of distinct agents — confirming H2 over the "agent interference" account.
  • Multi-agent repositories are common; direct agent-to-agent interactions are only partly observable (3 of 5 agents act through human GitHub accounts).
  • Results robust across four activity cutoffs (≥1, 5, 10, 20 agentic PRs) and within multi-agent subset.

Limitations

  • Observational design: variance partition, not controlled intervention — supports non-reducibility but cannot establish causation.
  • Stronger causal emergence measures (effective information, dynamical independence) require controlled interventions or fine-grained time series the public record does not provide.
  • Agent interactions are only partly observable: Codex, Cursor, and Claude Code act through the human operator's account, so attributed interactions are an undercount.
  • Codebase size and age released only as deciles tied to anonymized hash; models on binned and raw values agree.
  • Repository-level signal is necessary but not sufficient for emergence — other mechanisms could produce the same pattern.

Relevance to Agentic AI / LLM Agents

Directly challenges the dominant evaluation paradigm (individual agent benchmarks like SWE-bench) by showing that risk lives at the ecosystem level, not the agent level. Echoes findings in multi-agent systems that inter-agent failures dominate intra-agent ones. Has concrete implications for how AI coding tools (Codex, Devin, Copilot, Cursor, Claude Code) are evaluated, deployed, and governed in shared repositories — the metric that matters is repository health, not per-agent pass rates.