Skip to content

A Semantic-Layer-Mediated Agent for Natural Language to SQL over Heterogeneous Enterprise Databases

🕒 Published (v1): 2026-06-30 02:22 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper presents a NL2SQL agent that interposes a curated semantic layer and a compact intermediate representation (the Semantic Model Query, SMQ) between the LLM and raw enterprise database schemas, using deterministic SMQ→SQL compilation to supply verified physical identifiers. Driven by Gemini 3 Pro, it achieves 94.15% execution accuracy on the 547-task Spider2-snow benchmark, the third-highest leaderboard entry. The system is packaged as an end-to-end evaluation harness routing queries across SQLite, BigQuery, and Snowflake.

Problem

Enterprise NL2SQL is far harder than academic benchmarks: schemas have hundreds of opaque columns, queries demand CTEs, window functions, and dialect-specific syntax, and direct LLM prompting against raw schemas yields brittle SQL. The core difficulty has two components that naive prompting conflates: (1) grounding — identifying relevant tables/columns from a large, poorly self-describing schema; and (2) composition — assembling dialect-correct SQL once the right building blocks are known.

Method

The agent decouples grounding from composition via three mechanisms:

  1. Semantic layer: Each physical table is wrapped in a YAML semantic model exposing named dimensions, measures, and metrics, each with a human-readable description and an expr field holding the exact physical column expression. A per-database join graph declares typed join edges with on-predicates, so the agent never reverse-engineers joins.

  2. Semantic Model Query (SMQ): A compact JSON intermediate representation with three lists — metrics, filters, group_by — referencing semantic model elements by name. A deterministic engine compiles SMQs into dialect-correct SQL, resolving abstract names to physical expressions and injecting joins automatically.

  3. Constrained think–act loop: The agent runs a single-tool-per-step ReAct-style loop with three tools: getModelDataElements (schema discovery), convertSmqToSql (exploration; returns compiled SQL + 5-row preview), and execute (single terminal action). SMQ compilation is used for exploration only — the agent extracts verified physical identifiers from compiled fragments, then composes its own final SQL (adding window functions, recursive CTEs, etc.) using only names lifted from compiler outputs. Direct schema introspection (e.g., INFORMATION_SCHEMA) is forbidden.

The orchestration tier dispatches Spider2 instances through a thread pool with per-instance timeouts, retries, and backoff, keyed by instance ID; results are written to per-instance SQL/CSV files and scored with the official Spider2 evaluation suite. Configuration is centralized so the same harness runs across all three backends.

Key Contributions

  • System architecture separating schema grounding (via the semantic layer + SMQ) from SQL composition (agent-authored final query).
  • The SMQ intermediate representation: covers selection, filtering, grouping, and declared joins; deliberately omits advanced window functions and recursive CTEs so those are handled by agent composition on top of verified fragments.
  • Single-tool think–act agent loop with a strict "compile-to-explore, execute-once" discipline that constrains the action space and makes trajectories auditable.
  • Multi-backend evaluation harness routing SQLite/BigQuery/Snowflake by instance-ID prefix, scored under the official Spider2 protocol.
  • 94.15% execution accuracy on Spider2-snow (515/547), third-highest on the official leaderboard; analysis of per-class failure modes and the overfitting tension in semantic-layer curation.

Results

  • Overall: 94.15% execution accuracy on Spider2-snow (547 instances); 515/547 correct.
  • Per-backend breakdown:
  • SQLite-origin (sf_local): 131/135 = 97.0%
  • GA4-origin (sf_ga): 24/25 = 96.0%
  • BigQuery-origin (sf_bq): 350/369 = 94.9%
  • Native Snowflake (sf): 10/18 = 55.6% (smallest class, highest variance)
  • Leaderboard comparison (Spider2-snow execution accuracy):
  • DAIL-SQL + GPT-4o: 2.20%
  • Spider-Agent + o1-preview: 23.58%
  • Spider-Agent + Claude-4-Sonnet: 25.78%
  • ReFoRCE + o1-preview: 31.26%
  • This work (QUVI-3 + Gemini 3 Pro): 94.15% (3rd overall)
  • Extended thinking (Gemini 3 Pro, high setting) is critical; disabling it substantially degrades accuracy.

Limitations

  • SMQ covers only a common analytical core; the hardest queries rely on agent-authored SQL for which the engine provides no correctness guarantee.
  • Evaluation is single-benchmark (Spider2-snow); execution-result matching can over- or under-credit edge cases.
  • Per-backend accuracy for small classes (native Snowflake, 18 instances) is high-variance.
  • Accuracy is entangled with the maturity of each hand-curated semantic layer, which varies across databases — improving accuracy requires improving the layer, not just the agent.
  • Overfitting risk (Goodhart's Law): descriptions in the semantic layer can drift from reusable annotations toward question-specific hints, inflating benchmark accuracy while reducing generalization; no intrinsic quality metric exists to regularize this.

Relevance to Harnesses / Meta-Harnesses

This paper contributes a concrete end-to-end evaluation harness architecture: an orchestration tier managing parallel batched execution with retries/timeouts, per-instance workflow identifiers, multi-backend routing, and scoring integration with the official Spider2 suite — all configurable from a single application config with no code changes across backends. The "single terminal action" agent discipline (explore freely via SMQ, execute once) is a harness-level constraint enforced through prompting, not just a model choice, making it a design pattern for controllable agentic pipelines. The overfitting tension between semantic-layer quality and benchmark accuracy is directly analogous to the context-poisoning and reward-proxy problems that meta-harness designers face when constructing evaluation environments: curated context improves performance but can leak task-specific signal, raising questions about how to measure harness quality independently of downstream task accuracy.