Skip to content

SCARE: A Benchmark for SQL Correction and Question Answerability Classification for Reliable EHR Question Answering

🕒 Published (v1): 2025-11-13 06:35 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SCARE is a 4,200-instance benchmark for evaluating a post-hoc safety layer in EHR text-to-SQL systems, jointly testing SQL error correction and question answerability classification (answerable/ambiguous/unanswerable). Experiments across seven baseline methods reveal a fundamental trade-off: models that preserve correct SQL well tend to miss problematic inputs, and vice versa.

Problem

Existing benchmarks address either SQL correction or question answerability classification in isolation, leaving a gap for evaluating an integrated post-hoc verification component—a modular layer that inspects both the user question and the generated SQL before execution. Prior "abstention-aware" generation benchmarks conflate the generator's capability with the verifier's, making it impossible to assess post-hoc verification independently in safety-critical clinical settings.

Method

SCARE constructs 4,200 triples \((q, \hat{y}, o^*)\) across MIMIC-III, MIMIC-IV, and eICU. Answerable questions come from MIMICSQL/EHRSQL templates (GPT-4o paraphrasing for MIMIC-IV and eICU); ambiguous and unanswerable questions are generated by GPT-4o under six fine-grained categories (vague-question, vague-word, ambiguous-reference; small-talk, out-of-scope, missing-column) and then human-validated. Candidate SQL queries are generated by seven diverse text-to-SQL models (LLM-SQL, CodeS-15B, DIN-SQL, MAC-SQL, DeepSeek R1-70B, o4-mini, Qwen3-32B) to ensure realistic error distributions. The dataset is stratified to 350 instances per stratum (answerable-correct/incorrect/ambiguous/unanswerable) per database. Seven baseline strategies are evaluated: Two-Stage (decoupled classifier then SQL verifier), Single-Turn, Single-Turn-Veri (iterative retry loop), Multi-Turn-SelfRef (self-refinement), and three hybrid "-Cls" variants that inject the Two-Stage classifier's reasoning into the integrated models.

Key Contributions

  • First benchmark jointly evaluating post-hoc SQL correction and question answerability classification for EHR QA.
  • Fine-grained taxonomy of problematic questions: three ambiguity types (VQ, VW, AR) and three unanswerability types (ST, OS, MC).
  • Stratified 4,200-instance dataset grounded in three major EHR databases with candidate SQL from seven heterogeneous models.
  • Empirical characterization of the PR/CR vs. classification recall trade-off across modular, integrated, iterative, and hybrid architectures.

Results

  • Preservation Rate (PR): Single-Turn achieves the highest PR at 97.9%; Two-Stage falls to 80.4% due to misclassifying answerable questions.
  • Correction Rate (CR): Best single model is Single-Turn-Veri-Cls at 53.8%; iterative refinement (Multi-Turn-SelfRef) reaches 51.4% vs. 49.6% for basic Single-Turn.
  • Unanswerable F1: Two-Stage achieves 71.6% recall (93.0%) but poor precision; Single-Turn-Veri-Cls reaches F1 = 80.8% with better balance.
  • Ambiguous F1: All methods struggle; best is Single-Turn-Cls/Single-Turn-Veri-Cls at F1 ≈ 53.9–54.0%, with recall on vague-word (VW) never exceeding ~32%.
  • SQL error type breakdown: T/C reference errors are most correctable (CR 55.1% with Single-Turn-Veri/Multi-Turn-SelfRef); Other Global (OG) structural errors are hardest (CR 30.8%).
  • Hybrid "-Cls" variants dominate non-hybrid counterparts across almost all metrics.

Limitations

  • Balanced stratified sampling does not reflect the skewed real-world distribution of user queries.
  • Ambiguity detection remains weak across all methods (VW recall ≤ 32%), suggesting current LLMs lack the linguistic sensitivity for this sub-task.
  • Overall SQL correction rates are low (best CR ~54%), especially for globally structural errors.
  • Backbone evaluations primarily use Gemini-2.5-Flash; other models are relegated to appendices and may behave differently.
  • The benchmark uses demo (not full) versions of MIMIC-IV and eICU to avoid value-shuffling artifacts, which may limit generalizability to production schemas.

Relevance to Harnesses / Meta-Harnesses

SCARE directly instantiates the post-hoc verification layer pattern central to harness design: a modular, black-box-agnostic component that wraps an upstream generator and applies structured checks before execution. The benchmark's taxonomy of strategies—two-stage pipelines, single-turn integrated models, iterative retry loops, and hybrid classifier-injected variants—maps directly onto architectural patterns found in agentic meta-harnesses (pre-check, self-reflection, tool-gated execution). The finding that hybrid "-Cls" designs outperform both pure decoupled and pure integrated approaches provides empirical grounding for meta-harness design choices about when to use a specialized sub-agent versus an end-to-end pass with injected context.