Skip to content

Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Spider 2.0 is a benchmark of 632 real-world enterprise text-to-SQL workflow tasks requiring agents to navigate project codebases, cloud databases (BigQuery, Snowflake), and SQL dialect documentation—conditions that existing benchmarks ignore. The best code agent (o1-preview via Spider-Agent) solves only 21.3% of tasks, versus 91.2% on Spider 1.0 and 73.0% on BIRD. This exposes a wide capability gap between LLMs' benchmark performance and enterprise-grade SQL deployment.

Problem

Prior text-to-SQL benchmarks (Spider 1.0, BIRD) use small, synthetic schemas with simple SQL and pre-prepared inputs. Real enterprise workflows involve databases with thousands of columns across heterogeneous cloud/local systems, multiple SQL dialects, DBT project codebases, multi-step execution loops, and outputs exceeding 100 SQL lines—none of which prior benchmarks exercise.

Method

Benchmark construction: 632 tasks are sourced from real industrial databases (Google Analytics, Salesforce, Fivetran/DBT pipelines) across 6 database systems (BigQuery, Snowflake, SQLite, DuckDB, PostgreSQL, ClickHouse). Each task ships with a project codebase, database interface, and dialect documentation. Annotators rewrite collected SQLs at surface and semantic levels to prevent contamination (84.2% surface, 42% semantic rewrites). Evaluation uses execution-based focused assessment—scripts accept string, table, or database-file outputs and target only essential columns to reduce false negatives.

Agent framework (Spider-Agent): Inspired by ReAct and Intercode, Spider-Agent enables multi-turn, command-line interaction with the database environment, iteratively modifying SQL/Python until a final answer is obtained: \(A = O_k\) where \(O_k = \text{execute}(C, I, Q)\) at turn \(k\).

Variants: Spider 2.0-lite and Spider 2.0-snow strip the codebase and restrict output to SQL only, enabling comparison with classical text-to-SQL parsers; they share the same raw data but provide less runtime feedback than the full agentic setting.

Key Contributions

  • A 632-task benchmark grounded in real enterprise databases (avg. 812 columns/DB, avg. 144 tokens/SQL, avg. 7.1 advanced dialect functions/query)
  • An agentic task formulation: agent must discover schema, consult documentation, and iteratively execute code—not merely generate a single SQL given a pre-prepared schema
  • Spider-Agent: a code agent framework designed for database-centric multi-turn interaction
  • Two self-contained subsets (Spider 2.0-lite, Spider 2.0-snow) for parser-style baselines
  • Systematic error analysis identifying four failure modes: schema linking at scale, dialect handling, multi-step query planning, and codebase comprehension

Results

  • Spider 2.0 (agentic): o1-preview + Spider-Agent: 21.3% SR; DeepSeek-V3: 8.78%
  • Spider 2.0-lite (EX): o1-preview: 23.22%; o3-mini: 23.40%; Claude-3.5-Sonnet: 15.54%; GPT-4o: 13.16%; best traditional parser (DAIL-SQL + GPT-4o): 5.68%
  • Spider 2.0-snow (EX): o1-preview: 23.77%; DAIL-SQL + GPT-4o: 5.68%; SFT CodeS-15B: 0.73%
  • Contrast: same methods achieve 86.6% on Spider 1.0 and 57.4–66.7% on BIRD
  • Hard tasks (≥160 tokens) drop all models to ≤15% EX even on lite/snow settings

Limitations

  • Ground-truth evaluation requires live access to cloud databases (BigQuery, Snowflake), raising cost and reproducibility concerns
  • Success Rate is binary (0/1 per task), providing no partial-credit signal for near-correct multi-step solutions
  • The benchmark is English-only and tied to specific industrial database snapshots that may evolve
  • Spider 2.0-lite/snow provide less execution feedback than the full agentic setting, making direct difficulty comparisons across variants complicated
  • DBT project tasks (12.34% of Spider 2.0) require tool-specific knowledge that may disadvantage models lacking DBT fine-tuning

Relevance to Harnesses / Meta-Harnesses

Spider 2.0 is itself an evaluation harness for code agents: it wraps heterogeneous database environments, dialect documentation, and project codebases into a unified execution loop that scores agent behavior end-to-end—directly analogous to meta-harness design. Spider-Agent exemplifies a harness pattern where an LLM orchestrates multi-turn tool calls (SQL execution, shell commands, codebase inspection) and self-corrects via execution feedback, which is the core loop of agentic meta-harnesses. The benchmark's two-tier structure (full agentic vs. lite/snow parser mode) illustrates how harness designers must choose how much scaffolding to pre-provide versus expose as agent responsibility. Results here set a clear empirical floor for how far current harness-driven agents fall short on tasks requiring dynamic environment interaction rather than static prompt-to-output generation.