Survey on Evaluation of LLM-based Agents¶

🕒 Published (v1): 2025-03-20 17:59 UTC · Source: Arxiv · Venue: ACL · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This survey provides the first comprehensive taxonomy of evaluation methods for LLM-based agents, organized across five perspectives: core capability benchmarks, application-specific benchmarks (web, SWE, scientific, conversational), generalist agent leaderboards, structural benchmark dimensions, and developer-facing evaluation frameworks. It identifies a shift toward dynamic, realistic, continuously updated evaluations and flags critical gaps in cost-efficiency, safety, fine-grained metrics, and robustness assessment. The paper serves as a map of the current evaluation landscape rather than proposing a new method.

Problem¶

Static LLM benchmarks (text-in, text-out) are insufficient for agents that operate through sequential decision-making, multi-step tool use, and interaction with dynamic environments. There is no unified treatment of how agent evaluation has evolved, what structural dimensions benchmarks share, or how developer-facing evaluation frameworks differ from static benchmarks—leaving practitioners without a principled basis for choosing or designing evaluations.

Method¶

Literature survey structured across five orthogonal perspectives: (1) isolated capability benchmarks (planning, tool use, self-reflection, memory); (2) application-specific benchmarks reviewed per domain; (3) generalist agent leaderboards and cross-domain unification efforts; (4) taxonomic analysis of benchmarks along five structural dimensions—data curation strategy, environment dynamicity (static vs. dynamic), interaction interface (Code/Terminal, Tools, GUI), evaluation metric type, and safety coverage; (5) review of live developer evaluation frameworks. A comparative table (Table 1) encodes representative benchmarks across all five dimensions.

Key Contributions¶

First comprehensive survey mapping the full LLM-based agent evaluation landscape with a unified five-perspective structure.
Taxonomy of core benchmark dimensions (data, environment, interface, metric, safety) enabling cross-benchmark structural comparison.
Identification of critical open gaps: no standardized self-reflection benchmark, insufficient safety/robustness metrics (most benchmarks score 0 on safety column), lack of fine-grained intermediate-step metrics beyond binary task completion, and poor cost-efficiency measurement.
Documentation of the evolution from static, synthetic, single-step evaluations toward dynamic, real-world, multi-turn, MCP-sourced tooling benchmarks.
Review of agent evaluation frameworks (LangSmith, Langfuse, Galileo, Vertex AI, AutoGen, etc.) as a distinct category from static benchmarks, noting their role in continuous dev-cycle monitoring.
Identification of Harbor and Exgentic as the first cross-environment unified evaluation protocols for generalist agents.

Results¶

The paper is a survey; it reports no new experimental numbers. Cited headline findings from reviewed works: - SWE-bench Pro: models achieve below 25% Pass@1 on 1,865 human-verified, multi-file tasks. - SWE-Lancer: 1,400 Upwork tasks with >$1M in payouts; exposes gaps in long-term reasoning. - GAIA and OSWorld: performance has improved but harder subsets remain challenging for all current agents. - BFCL v2/v3 and ComplexFuncBench: despite model advances, multi-turn, multi-step tool-use benchmarks continue to expose failures. - Even SOTA models "struggle with long-horizon planning" per PlanBench-class evaluations.

Limitations¶

Survey is necessarily incomplete; the GitHub tracker is cited for ongoing coverage but the paper itself is a snapshot.
Section 6 on frameworks is cut off in the provided text, leaving the framework analysis (beyond observability and final-response evaluation) incomplete.
No meta-analysis of benchmark correlations or predictive validity (i.e., whether benchmark scores predict real-world deployment performance).
Safety dimension is flagged as critical yet only τ-Bench among Table 1 benchmarks explicitly incorporates safety constraints—the survey does not resolve this gap, only surfaces it.
Focus on four application domains (web, SWE, scientific, conversational) excludes embodied, game, and search agents despite acknowledging their existence.

Relevance to Harnesses / Meta-Harnesses¶

The paper explicitly frames LLM-based agents as "backbone LLM + agent harness" (citing Yao et al., 2022b), making the harness concept a foundational architectural primitive in this entire evaluation taxonomy. Section 6 surveys evaluation frameworks (LangSmith, Langfuse, Galileo, AutoGen, Vertex AI) that function as meta-harnesses—wrapping arbitrary agent harnesses with continuous monitoring, trajectory logging, LLM-judge evaluation, and regression testing infrastructure. The emergence of Harbor Framework and Exgentic as unified cross-environment evaluation protocols represents the first attempt at a true meta-harness standard: a single protocol capable of running any agent harness across any benchmark environment, which directly addresses the fragmentation problem that harness researchers face. This survey is therefore the most comprehensive reference available for understanding what evaluation infrastructure exists around agent harnesses and where the gaps are.