Measuring Agents in Production¶

🕒 Published (v1): 2025-12-02 16:45 UTC · Source: Arxiv · Venue: ICML 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

MAP is the first large-scale empirical study of LLM agents in production, combining 20 in-depth interviews and 86 deployed-system survey responses across 26 domains. The central finding is that production agents overwhelmingly favor simplicity and control—short workflows, prompt engineering over fine-tuning, and heavy human-in-the-loop evaluation—diverging sharply from current research trends. Reliability, not capability, is the dominant engineering constraint driving these choices.

Problem¶

Despite widespread excitement and deployment of LLM agents, virtually no systematic, first-hand technical data exists on how production agents are actually built, evaluated, and operated. Published academic work surveys research prototypes; company blog posts cover individual systems. This leaves the research community blind to the deployment realities that determine whether agents succeed or fail in practice.

Method¶

Mixed-method empirical study conducted April–November 2025: - Case studies: 20 semi-structured interviews (30–90 min) with ML/software engineers from startups to global enterprises, spanning 5 sectors. Analyzed using grounded theory open coding with ≥3 independent coders per interview. - Survey: 47-question Qualtrics survey with dynamic branching distributed through AI summits, MOOCs, and professional networks; 306 responses filtered to 86 deployed/pilot systems. - Four research questions (RQ1–4) cover applications, architecture/methods, evaluation, and challenges. Bootstrap resampling (N=1,000) provides 95% CIs for categorical comparisons.

Key Contributions¶

First systematic, practitioner-sourced empirical study of production LLM agents at scale (86 deployed systems, 20 case studies).
Quantitative characterization across 17 design dimensions: model selection, weight tuning, prompting, architecture, evaluation methods, latency, security.
Documents a reproducible central principle: reliability is achieved via system-level design (constrained workflows, HITL, read-only modes), not algorithmic advances.
Surfaces concrete research-practice gaps: latency tolerance enabling inference-time scaling, multi-model orchestration needs, benchmark scarcity, post-training brittleness under model upgrades.

Results¶

Motivation: 80% cite productivity increase; 72% cite reduced human task-hours as primary drivers.
Model choice: 17/20 case studies use closed-source frontier models (Claude, GPT-o3); 70% use no weight tuning (off-the-shelf prompting).
Prompt construction: 79% rely on manual or manual+LLM drafting; only 9% use automated prompt optimizers (e.g., DSPy).
Architecture: 68% execute ≤10 steps before human intervention; 80% use structured (predefined) workflows rather than open-ended planning.
Frameworks: 85% of production case studies use custom in-house implementations; only 3/20 use LangChain/LangGraph or DSPy.
Evaluation: 74% rely primarily on human-in-the-loop evaluation; 75% forgo formal benchmarks entirely, using A/B testing and expert feedback instead.
Latency: 66% tolerate minute-scale or longer response times; only 5/20 cases require real-time responsiveness.
Top challenge: 38% rank core technical performance (reliability/robustness) as top priority, far exceeding governance (3%) or compliance (17%).

Limitations¶

Geographic concentration: case study teams skew toward the Americas with few European participants.
Participation bias: survey respondents likely over-represent professional networks of the research team; interview participants self-selected based on company policy.
Temporal snapshot: data covers April–November 2025 in a rapidly evolving field; fine-grained prevalence estimates may shift.
No direct system auditing—all data is self-reported by practitioners; claims about reliability or scale are unverified externally.
Survey N=86 deployed systems is meaningful but not comprehensive global coverage.

Relevance to Agentic AI / LLM Agents¶

MAP is a rare empirical anchor for a field dominated by benchmark-driven research: it directly contradicts several assumptions implicit in academic agent work—that more autonomy, RL training, and automated orchestration are the right directions—by showing practitioners deliberately avoid these in favor of constrained, human-supervised pipelines. For researchers tracking agentic AI, the study highlights concrete underexplored problems: evaluation tooling for long-horizon non-verifiable tasks, post-training methods robust to model version churn, and principled human-in-the-loop architectures rather than treating HITL as a temporary scaffold. The finding that 66% of deployments tolerate minute-scale latency also opens space for inference-time compute scaling and test-time verification architectures that sacrifice speed for correctness.