BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation via Lens of Dynamic Interactions

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Dynamic multi-turn text-to-SQL benchmark with interaction, errors, and ambiguity handling

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

BIRD-INTERACT is a dynamic multi-turn text-to-SQL benchmark that replaces static conversation transcripts with a live interactive environment featuring a function-driven user simulator, hierarchical knowledge base, and full CRUD task coverage. It exposes a severe gap in current LLM capability: even GPT-5 solves only 8.67% of tasks under the protocol-guided setting and 17.00% under the agentic setting. The work also introduces Interaction Test-time Scaling (ITS), showing performance improves monotonically with more interaction budget.

Problem

Existing multi-turn text-to-SQL benchmarks evaluate models against fixed, pre-recorded dialogue transcripts (static context), failing to reward or penalize actual interaction strategies. They also restrict tasks to SELECT-only queries, ignoring INSERT/UPDATE/DELETE/DDL operations prevalent in real database workflows. This means no prior benchmark captures the full interactive loop of ambiguity resolution, error recovery, and evolving user intent that characterizes production database assistants.

Method

BIRD-INTERACT is built on top of LiveSQLBench and contributes three core components:

1. Interactive environment: Each of 900 tasks is paired with an executable PostgreSQL sandbox, metadata files, and a Hierarchical Knowledge Base (HKB) organized as a DAG to model multi-hop knowledge dependencies.

2. Two-stage function-driven user simulator: In stage 1, an LLM semantic parser maps the system's clarification request to one of three symbolic actions — \(\text{AMB}()\) (annotated ambiguity resolution), \(\text{LOC}()\) (AST-based SQL fragment retrieval for unannotated but reasonable requests), or \(\text{UNA}()\) (rejection of ground-truth-eliciting queries). Stage 2 generates the natural-language response conditioned on the chosen action and the annotated gold SQL snippet. This two-stage design prevents ground-truth leakage and task drift.

3. Two evaluation settings:

4. c-Interact: Protocol-guided dialogue with budget \(\tau_\text{clar} = m_\text{amb} + \lambda_\text{pat}\) clarification turns; each sub-task allows one debugging attempt after execution failure.
5. a-Interact: ReAct-style agentic setting exposing 9 callable tools (schema retrieval, knowledge lookup, SQL execution, user query, etc.) with a budget \(B = B_\text{base} + 2m_\text{amb} + 2\lambda_\text{pat}\), where \(B_\text{base} = 6\).

Ambiguity injection converts single-turn LiveSQLBench tasks into interactive ones via: (i) superficial user query ambiguities (intent-level and implementation-level), (ii) knowledge ambiguities (one-shot removal or chain-breaking of intermediate HKB nodes), and (iii) environmental ambiguities (e.g., NULLs in critical fields). Each task has two sequential sub-tasks; sub-task \(q_2\) is released only after \(q_1\) is successfully completed, enforcing state dependency.

Key Contributions

• BIRD-INTERACT-FULL (600 tasks, up to 11,796 dynamic interactions) and BIRD-INTERACT-LITE (300 tasks, simplified DBs) covering full CRUD spectrum across BI and DM domains.
• A two-stage function-driven user simulator that prevents ground-truth leakage and task deviation, with inter-annotator agreement of ~93.4%.
• Two complementary evaluation protocols: conversational (c-Interact) and fully agentic (a-Interact) with budget-constrained awareness.
• Executable test-case-based correctness verification (functional equivalence, not string match).
• Empirical discovery of Interaction Test-time Scaling (ITS): performance improves monotonically as interaction budget increases.
• Memory grafting analysis validating the importance of effective interaction history management.

Results

• c-Interact (FULL): Gemini-2.5-Pro leads at 16.33% overall SR and 20.92 normalized reward; GPT-5 achieves only 8.67% SR. DM tasks are substantially harder than BI tasks (e.g., GPT-5: 25.40% on DM priority vs. 9.49% on BI priority).
• a-Interact (FULL): GPT-5 reaches 17.00% SR (30.00 NR); Gemini-2.5-Pro achieves 29.17% SR (25.52 NR); Claude-Sonnet-4 achieves 27.83% SR.
• Debugging adds marginal gain in c-Interact (e.g., +2.50% for Gemini-2.5-Pro overall).
• a-Interact outperforms c-Interact across all models, suggesting structured protocol is more restrictive than agentic freedom for current models.
• Models exhibit a trial-and-error bias in a-Interact (preferring SQL execution over strategic knowledge/user consultation).
• All models score substantially below human-level, leaving large headroom for future work.

Limitations

• Single run per model due to cost constraints; no confidence intervals reported.
• Budget parameters (\(\lambda_\text{pat} = 3\), \(B_\text{base} = 6\)) are fixed defaults; sensitivity analysis is limited.
• The simulator's symbolic action space (AMB/LOC/UNA) may not cover all real-world clarification patterns.
• Tasks are capped at \(n = 2\) sub-tasks; longer chains may reveal additional failure modes.
• Benchmark is English-only and PostgreSQL-specific.
• User simulator itself incurs ~$0.03 per task cost, making large-scale ablations expensive.

Relevance to Agentic AI / LLM Agents

BIRD-INTERACT directly benchmarks LLMs as interactive agents operating within a tool-augmented environment — clarifying ambiguity, executing SQL, consulting knowledge bases, and recovering from errors — which is the canonical agentic loop. The a-Interact setting using the ReAct paradigm with 9 discrete tools serves as a rigorous stress test for autonomous planning and resource allocation under budget constraints, directly relevant to the broader challenge of making LLM agents reliable in long-horizon, stateful tasks. The ITS finding parallels test-time compute scaling in reasoning, suggesting that allocating more interaction budget (rather than just more compute per token) is a promising axis for agentic improvement. The function-driven simulator design also addresses a key evaluation bottleneck for agent research: how to benchmark agents at scale without human-in-the-loop supervision while preventing reward hacking.