Skip to content

Agentic Frameworks for Reasoning Tasks: An Empirical Study

🕒 Published (v1): 2026-04-17 19:02 UTC · Source: Arxiv · link

Why this paper was selected

Empirical comparison of agentic frameworks across reasoning tasks and efficiency

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper empirically evaluates 22 agentic frameworks on three reasoning benchmarks (BBH, GSM8K, ARC) under a unified experimental setting, measuring accuracy, latency, and cost. It finds that framework failures are predominantly orchestration-level (uncontrolled context growth, retry loops, quota exhaustion) rather than reasoning-level, and that mathematical reasoning degrades sharply across all architectures.

Problem

Despite rapid proliferation of agentic frameworks, there is no comprehensive, large-scale empirical comparison that benchmarks reasoning accuracy, cost, latency, and cross-benchmark consistency simultaneously. Practitioners lack evidence to guide framework selection beyond architectural descriptions or claimed capabilities.

Method

1,200 GitHub repositories collected between January 2023 and July 2025 were filtered to 55 candidates via inclusion/exclusion criteria (activity, stars/forks, issue health, English docs), then ranked to yield 22 frameworks. Each framework was wrapped in a unified harness and run against: BBH (complex multi-step reasoning), GSM8K (math word problems), and ARC (science QA). The 22 frameworks were taxonomized into five architectural categories: single-agent, role-based multi-agent, hierarchical, modular, and graph-based. Per-task accuracy, wall-clock execution time, and API cost (in USD) were logged for analysis. Consistency was assessed by comparing each framework's performance rank across the three benchmarks.

Key Contributions

  • Systematic selection pipeline from 1,200 GitHub repositories to 22 frameworks, with a five-category architectural taxonomy.
  • Unified experimental harness enabling apples-to-apples comparison of reasoning accuracy, cost, latency, and cross-benchmark consistency.
  • Identification of orchestration failure modes (context growth, retry storms, quota exhaustion) as the primary differentiator between high- and low-performing frameworks.
  • Public dataset, per-task logs, and source code released for replication.

Results

  • 19/22 frameworks completed all three benchmarks; 3 failed to finish (Camel, Upsonic, partially AutoGen/Mastra).
  • Consistent performers (12 frameworks): mean accuracy 74.6–75.9%, execution time 4–6 s/task, cost 0.14–0.18¢/task.
  • GSM8K accuracy gap: mean 44.35% on GSM8K vs. 89.80% on BBH and 89.56% on ARC — a ~45-point drop consistent across all architectures.
  • Orchestration failures at scale:
  • Camel: failed to complete BBH after 11 days due to uncontrolled context growth.
  • Upsonic: consumed $1,434 in one day from extraction-failure retries amplified by token accumulation.
  • AutoGen and Mastra: exhausted API quotas through iterative interactions that grew prompt length without improving answers.

Limitations

  • All experiments use a single base LLM (model not explicitly named in the excerpt), so results may not generalize across model families.
  • Framework configurations were left at defaults where possible; tuned configurations might close gaps between consistent and inconsistent performers.
  • GSM8K degradation is attributed to base-model weaknesses, but the study does not ablate whether tool-augmented math execution (code interpreter) would change outcomes.
  • The taxonomy categories are coarse; real frameworks blend multiple patterns and the mapping may not capture hybrid designs accurately.
  • Benchmarks (BBH, GSM8K, ARC) are standard NLP tasks, not end-to-end software engineering pipelines, limiting direct transfer of findings to production contexts.

Relevance to Harnesses / Meta-Harnesses

This study is itself a meta-harness exercise: the authors built a unified orchestration wrapper to drive 22 heterogeneous agentic frameworks through the same benchmark pipeline, measuring cost and correctness — exactly the concern of anyone building or evaluating a paper-digest or experiment meta-harness. The central finding — that orchestration quality (memory discipline, retry policy, context budgeting) dominates reasoning quality — directly informs harness design: a meta-harness must enforce hard context limits, bounded retry logic, and per-task cost caps to prevent runaway behaviour analogous to Camel's 11-day run or Upsonic's $1,434 day. The taxonomy of five architectural categories (single-agent through graph-based) also provides a structured vocabulary for classifying which sub-agents a meta-harness should compose and what failure modes to guard against for each type.