Skip to content

CryptoBench: A Dynamic Benchmark for Expert-Level Evaluation of LLM Agents in Cryptocurrency

🕒 Published (v1): 2025-11-29 09:52 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CryptoBench is a live, expert-curated benchmark of 50 monthly questions evaluating LLM agents on real-world cryptocurrency analysis tasks. It exposes a critical retrieval–prediction imbalance: leading models handle data retrieval adequately but nearly fail at predictive reasoning. Incorporating an agentic framework (SmolAgent + web browser tool) reshuffles model rankings, confirming that raw model quality does not transfer directly to agentic execution.

Problem

Existing agentic benchmarks (WebArena, GAIA, AgentBench) rely on static environments, sandboxed data, or historical facts, making them unsuitable for cryptocurrency analysis—a domain defined by real-time adversarial information, continuously evolving protocols, and data scattered across specialized on-chain platforms, DeFi dashboards, and derivatives feeds. No prior benchmark combined live data access, domain-expert task design, and dynamic monthly refresh.

Method

Tasks are built via a templated question pool: experts draft questions with parameterized entities (token addresses, wallet hashes, dates), a three-stage adversarial validation pipeline (Author → Validator → Adjudicator) gates inclusion, and monthly re-instantiation of templates with fresh entities provides contamination resistance.

The four-quadrant classification partitions tasks by two axes: cognitive demand (Retrieval vs. Prediction) and operational complexity (Simple vs. Complex), yielding SR / CR / SP / CP categories (Figure 2 distribution: 34% / 34% / 12% / 20%).

Evaluation runs each of ten frontier LLMs in two modes: (1) direct prompting, and (2) inside SmolAgent equipped with a single web-browsing tool (navigate, search, extract). Responses are scored 0–3 via an LLM-as-a-Judge rubric, and the Average Success Rate is

\[\text{ASR} = \frac{\sum_{i=1}^{N} \text{score}_i}{N \times 3} \times 100\%\]

with a ±5% numerical tolerance for volatile market values.

Key Contributions

  • First expert-curated, live cryptocurrency benchmark supporting monthly dynamic updates and contamination resistance via entity-parameterized templates.
  • Four-quadrant diagnostic framework (SR/CR/SP/CP) enabling fine-grained capability profiling of agents.
  • Broad real-world platform coverage: on-chain intelligence aggregators, DeFi dashboards, DEX analytics, and derivatives data—no sandboxing.
  • Empirical finding that agentic framework deployment (SmolAgent) substantially reshuffles model rankings relative to direct LLM evaluation, isolating planning and tool-use as independent capability axes.
  • Demonstration of a systematic retrieval–prediction imbalance across ten state-of-the-art frontier models.

Results

  • Best direct-LLM score: ~44.0% ASR (Grok-4 Web); most models cluster in the 12–30% range.
  • Best SmolAgent score: ~30.0% ASR; most agentic scores are lower than direct scores for the same model, with re-ordered rankings.
  • Models competent on SR/CR exhibit near-complete failure on SP/CP quadrants.
  • Agentic execution introduces additional failure modes (planning errors, tool misuse, error propagation) that reduce overall accuracy despite enabling real-time data access.
  • Evaluated models include Grok-4, GPT-5, Qwen3-Max, Claude 4.1 Opus, Claude 4.5 Sonnet, DeepSeek R1, Gemini 2.5 Pro, GPT-OSS 120B (evaluated Oct 12–Nov 11 window).

Limitations

  • Benchmark covers only 50 questions per monthly cycle—small sample size limits statistical power for per-quadrant analysis.
  • Agentic framework is restricted to a single generic web-browsing tool; specialized API access (e.g., on-chain RPC, exchange APIs) is excluded, potentially underestimating what production systems could achieve.
  • LLM-as-a-Judge evaluation introduces its own failure modes and is not validated against human judge agreement in the paper text provided.
  • Monthly update frequency may still lag the fastest-moving market narratives; template refresh scope is not detailed beyond entity substitution.
  • Coverage is cryptocurrency-only; generalizability to broader financial domains is unverified.

Relevance to Harnesses / Meta-Harnesses

CryptoBench's templated pipeline (draft → adversarial validate → adjudicate → monthly re-instantiate) is itself a domain-specific meta-harness for benchmark generation, closely analogous to harness architectures that orchestrate agent evaluation pipelines rather than static test suites. The two-mode evaluation design—direct model vs. SmolAgent-wrapped model—directly probes how a harness layer (tool scaffolding, multi-step planning) transforms raw model capability, which is a core question in meta-harness research. The finding that agentic wrapping reshuffles rankings without uniformly improving scores is actionable evidence for harness designers about the non-trivial cost of scaffolding. For researchers building or studying harnesses, CryptoBench provides a live, adversarial testbed where harness design choices (tool selection, error recovery, source routing) are first-class performance variables rather than implementation details.