Skip to content

An Empirical Study of Agent Developer Practices in AI Agent Frameworks

🕒 Published (v1): 2025-12-01 17:52 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper conducts the first large-scale empirical study of LLM-based agent frameworks, mining 1,575 GitHub projects and 20,620 developer discussions to characterize framework adoption patterns, classify development challenges across the SDLC, and compare ten frameworks across five developer-need dimensions. It reveals that multi-framework composition dominates real-world agent development and that performance optimization and version compatibility are universal pain points.

Problem

Agent frameworks have proliferated (100+ on GitHub, 400k+ stars) but their practical impact on developer workflows is poorly understood. Over 80% of developers struggle to select appropriate frameworks, yet no systematic study characterizes how frameworks are actually used, what recurring failures emerge, or how frameworks comparatively satisfy developer needs across the full SDLC.

Method

Three-phase empirical study using mining software repositories (MSR) methodology:

  1. Dataset construction: 1,575 GitHub agent projects (filtered by keyword, documentation completeness, ≥10 stars, ≥5 forks, 2022–2025) yield 8,710 cleaned discussion threads for RQ1/RQ2; 11,910 framework-specific discussions collected for RQ3 across ten identified frameworks.
  2. Framework identification: Topic-tag frequency analysis on 1,575 repos identifies ten representative frameworks (LangChain, LangGraph, AutoGen, CrewAI, MetaGPT, LlamaIndex, Swarm, BabyAGI, Camel, Semantic Kernel); functional roles extracted via a three-stage pipeline—TF-IDF keyword extraction → GPT-4o semantic classification → frequency analysis.
  3. Challenge taxonomy: Open-card sorting of 8,710 discussions into SDLC stages, producing four domains and nine subcategories.
  4. Comparative evaluation: Five-dimensional framework scoring (development efficiency, functional abstraction, learning cost, performance optimization, maintainability) derived from 11,910 framework-specific discussions.

Key Contributions

  • First large-scale empirical study of ten widely-used LLM agent frameworks using real developer discussions at scale.
  • Taxonomy of agent development challenges across the SDLC: four domains (Logic, Tool, Performance, Version) with nine subcategories.
  • Five-dimensional evaluation metric comparing frameworks against actual developer needs.
  • Empirical evidence that multi-framework composition (rather than single-framework use) is the dominant real-world development pattern.
  • Actionable findings and design implications for framework designers and agent developers.

Results

  • Challenge distribution: Logic failures 25.6% (dominated by task termination/infinite loop issues), Performance failures 25% (context retention, dialogue history loss, memory management), Version failures >23% (compatibility conflicts causing build failures), Tool failures 14% (API limits, permission errors, library mismatches).
  • Framework comparisons:
  • LangChain and CrewAI lowest learning cost (best for beginners).
  • AutoGen and LangChain best for rapid prototyping.
  • AutoGen and LangChain lead in task decomposition and multi-agent collaboration (functional abstraction).
  • Performance optimization is a universal shortcoming across all ten frameworks.
  • AutoGen and LangChain have the highest maintenance complexity despite mature ecosystems.
  • Discussion volume by framework: LangChain 6,006; LlamaIndex 1,691; LangGraph 978; AutoGen 988; BabyAGI 403; Swarm 391; Semantic Kernel 668; MetaGPT 312; Camel 249; CrewAI 224.

Limitations

  • Discussion-based data introduces self-selection bias: only issues developers chose to post publicly are captured; silent failures and internal team workarounds are invisible.
  • GPT-4o used for semantic classification of discussions introduces potential LLM classification errors not independently validated at scale.
  • The ten selected frameworks are top-frequency by GitHub topic tags and may not represent the full diversity of agent framework designs.
  • Snapshot data through July 2025; rapidly evolving frameworks may have changed substantially.
  • Challenge taxonomy is derived from discussion text rather than systematic issue-tracker or crash-report analysis, limiting precision.

Relevance to Harnesses / Meta-Harnesses

This study directly characterizes the practitioner-level failure modes of agent orchestration frameworks—the very systems that harnesses and meta-harnesses are built atop or designed to replace. The finding that task termination control (25.6% of logic failures) and performance/memory management (25%) are the dominant pain points provides empirical grounding for why meta-harness designs need explicit loop-control primitives and memory lifecycle management. The dominance of multi-framework composition as a development pattern suggests that meta-harnesses operating across heterogeneous framework stacks (rather than wrapping a single framework) reflect real-world usage. The five-dimensional evaluation schema (efficiency, abstraction, learning cost, performance, maintainability) offers a reusable rubric for evaluating meta-harness designs.