Skip to content

Spider 2.0-AIFunc: Extending Real-World Text-to-SQL to AI-Native SQL Workflows

🕒 Published (v1): 2026-07-07 12:54 UTC · Source: Arxiv · link

Why this paper was selected

AI-native SQL workflows extending agentic tool use to cloud data platforms — real deployment setting

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Spider 2.0-AIFunc is a benchmark of 465 verified instances that extends real-world text-to-SQL evaluation to cover AI-native SQL workflows, where target queries embed Snowflake Cortex AI function calls (e.g., AI_CLASSIFY, AI_SENTIMENT) alongside conventional SQL operators. Construction and verification are themselves carried out by an agent-based pipeline that rewrites source tasks, executes candidates against live databases, and iterates on errors. Top proprietary models reach 67–70% execution accuracy; agent frameworks designed for classical text-to-SQL do not transfer, with a minimal baseline matching or beating three elaborated alternatives.

Problem

Existing text-to-SQL benchmarks (WikiSQL, Spider 1.0, BIRD, Spider 2.0) evaluate only conventional SQL over structured data. Major cloud platforms now expose LLM inference as native SQL functions, creating a class of AI-native SQL workflows that mixes semantic operations with standard relational operators—a gap no existing benchmark measures. Two new challenges arise: specification determinism (instructions must fully constrain AI function parameters such as classification label sets) and execution determinism (LLM-backed functions can produce slightly varying outputs even at temperature zero, undermining execution-accuracy evaluation).

Method

Benchmark construction pipeline. Starting from 513 Spider2-Snow source instances, an agent (Claude Opus 4.5, ≤15 rounds/task) simultaneously rewrites the gold SQL to inject Cortex AI function calls and revises the natural-language instruction to match. The agent executes candidate queries live, repairs errors iteratively, and verifies that all AI function parameters are fully specified. A two-round scheme (main round: unrestricted function choice; diversity round: explicit priority to underrepresented functions) balances coverage across six function types.

Determinism verification. Each candidate undergoes four consecutive re-execution passes (5 + 10 + 10 + 10 runs, ≥35 total), with a repair agent that diagnoses inconsistencies via diagnostic SQL before adjusting the query, instruction, or comparison logic. An instance must achieve 100% identical results within every pass. A final stability check (10 runs × 3 temporally separated windows) confirms cross-session stability without further repair; failures are dropped.

Evaluation. Execution accuracy requires both the gold SQL \(S_n\) and predicted SQL \(\hat{S}_n\) to be run in the same time window \(t\): $\(\text{EX} = \frac{1}{N}\sum_{n=1}^{N} \mathbf{1}\!\left(\text{exec}(S_n,t),\, \text{exec}(\hat{S}_n,t)\right)\)$ Comparison uses column-subset matching (every gold column must appear in predicted output; extra predicted columns are penalized as absent). Ten models are evaluated via Spider-Agent with a minimal three-tool setup (bash, SQL execution, termination) plus AI function reference documentation.

Key Contributions

  • Spider 2.0-AIFunc benchmark: 465 verified instances across 125 databases, covering six Snowflake Cortex AI function types, derived from enterprise-grade Spider2-Snow queries averaging 65 lines and 2,617 characters.
  • Agent-based construction pipeline: reusable two-round pipeline (main + diversity) that rewrites both SQL and instructions, resolves specification ambiguity, and is generalizable to other platforms/functions.
  • Multi-pass determinism verification protocol: ≥35 in-session executions plus 30 cross-window executions per instance to guard against LLM non-determinism in SQL-embedded inference.
  • Empirical finding on framework transferability: minimal Spider-Agent consistently matches or outperforms three specialized text-to-SQL agent frameworks (AutoLink, ReFoRCE, DSR-SQL) on AI-native SQL.
  • Taxonomy of AI-native SQL error modes: predicate specification, schema grounding, and AI function parameterization identified as primary failure drivers.

Results

  • Claude Opus 4.6: 70.3% EX (99.4% exec.), highest overall.
  • Claude Sonnet 4.6: 69.0% EX; Gemini 3.1 Pro: 67.1% EX; GPT-5.4: 63.0% EX; Gemini 3 Flash: 60.9% EX.
  • Best open-source model (Kimi K2.5): 58.1% EX; remaining open-source models: 44.9–57.0% EX.
  • Agent framework comparison (all using Claude Sonnet 4.6 backbone):
  • Spider-Agent (minimal): 69.0% EX
  • AutoLink: 68.6% EX
  • ReFoRCE: 68.5% EX
  • DSR-SQL: 62.2% EX (despite 100% exec. rate)
  • Two-thirds of tasks (66.0%) require combining two AI functions within a single query.
  • Median execution time 4.3 s; 2.2% of instances exceed 60 s due to AI function fan-out over large row sets.

Limitations

  • Benchmark is Snowflake-platform-specific; coverage of BigQuery, Databricks, or other AI-SQL ecosystems is absent.
  • AI function reference documentation is provided at inference time, giving models an advantage over real-world scenarios where analysts must discover function APIs themselves.
  • LLM non-determinism at temperature zero is addressed by filtering rather than root-cause elimination; future model updates to Snowflake Cortex could invalidate stored gold outputs.
  • Construction and verification costs are substantial (≥35 executions/instance × live Snowflake), limiting scale and community reproducibility.
  • Qualitative error analysis is based on disagreement inspection rather than systematic annotation, leaving error-mode frequencies unreported.
  • Source databases excluded for non-static snapshots may introduce selection bias toward simpler or more static domains.

Relevance to Harnesses / Meta-Harnesses

The construction pipeline itself is a meta-harness: an agent-driven orchestration layer that wraps a live database environment, runs candidate SQL, detects failures, and iterates on both code (SQL) and specification (natural-language instructions) until quality gates are met—exactly the pattern of a test-and-repair harness operating at dataset-generation level. The multi-pass determinism verification is a second-order harness running atop the first, enforcing stability guarantees across temporally separated execution windows. The empirical finding that minimal harnesses match elaborate ones on AI-native SQL challenges the conventional wisdom that more scaffolding monotonically improves agentic performance, which is directly actionable for meta-harness designers choosing how much framework complexity to invest in.