Skip to content

Rethinking Agentic Workflows: Evaluating Inference-Based Test-Time Scaling Strategies in Text2SQL Tasks

🕒 Published (v1): 2025-10-13 01:29 UTC · Source: Arxiv · Venue: COLM 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper benchmarks six lightweight agentic workflows combining different test-time scaling strategies (Divide-and-Conquer, few-shot, parallel sampling, result verification, retrieval) across four LLMs on the BIRD Mini-Dev Text2SQL benchmark. The central finding is that Divide-and-Conquer prompting with 3-shot demonstrations reliably improves accuracy for both general and reasoning models, while more complex workflows yield inconsistent gains. Base model choice is at least as important as workflow design.

Problem

It is unclear which inference-time scaling strategies actually pay off when deploying Text2SQL systems in industry settings—particularly now that reasoning models (o4-mini, Gemini 2.5 Flash) have internalized multi-step reasoning via post-training. Prior work proposes increasingly complex agentic pipelines without systematic cost-accuracy-latency analysis under realistic constraints.

Method

Six agentic workflows are defined using a standardized notation (SW=SQL Writer, EX=Executor, SR=SQL Refiner, etc.) and evaluated on BIRD Mini-Dev (500 question-SQL pairs) across four LLMs: Gemini 1.5 Flash, Gemini 2.5 Flash, GPT-4o, o4-mini.

The six workflows in complexity order: 1. Baseline CoT+ReAct: \(\text{SW} \to \text{EX} \leftrightarrow \text{SR}\) 2. DC+ReAct: adds Divide-and-Conquer decomposition instruction 3. DC 3-shot+ReAct: adds 3-shot demonstrations to DC 4. DC 3-shot ×5 Parallel: five parallel DC 3-shot threads with majority vote; a Candidate Selector breaks ties 5. DC 3-shot+Verification: adds a Feedback Provider (FP) that checks semantic correctness of syntactically valid SQL 6. Retrieval+DC 3-shot: prepends \(\text{KE} \to (\text{ER} \| \text{CR}) \to \text{SW} \to \text{EX} \leftrightarrow \text{SR}\) using LSH-based entity retrieval and semantic column retrieval

Metrics: Soft-F1 (primary), Execution Accuracy (EX), R-VES, inference latency (seconds), prompt tokens (PT), and completion tokens (CT).

Key Contributions

  • Standardized notation and taxonomy for Text2SQL agentic workflow components drawn from prior literature (Table 1–2)
  • Systematic cost-accuracy-latency profiling of six workflows × four LLMs on a common benchmark
  • Empirical evidence that DC + few-shot demonstrations benefit reasoning models as well as general-purpose models
  • Finding that retrieval-enhanced schema filtering is counterproductive when queries require complex schema reasoning, increasing Wrong Query Logic and Schema Linking Error rates
  • Quantification of the latency asymmetry: incorrect answers take 19.58% longer to generate than correct ones

Results

  • Best single workflow: DC 3-shot+ReAct; Soft-F1 gains over CoT+ReAct baseline: +5.1 (Gemini 1.5 Flash: 57.51→62.58), +2.4 (Gemini 2.5 Flash: 65.75→68.12), +3.4 (GPT-4o: 61.07→64.44), +9.3 (o4-mini: 56.27→65.53)
  • Verification step: adds modest consistent gains for 3/4 models (Gemini 2.5 Flash: 68.12→68.28, GPT-4o: 64.44→64.95, Gemini 1.5 Flash: 62.58→63.63); neutral for o4-mini
  • Parallel ×5: marginal or negative gains (Gemini 2.5 Flash drops from 68.12 to 66.00 Soft-F1)
  • Retrieval: underperforms DC 3-shot+ReAct on nearly every model; o4-mini Soft-F1 drops from 65.53→64.58
  • Base model dominance: Gemini 2.5 Flash baseline (65.75 Soft-F1) exceeds the best GPT-4o workflow (64.95) and best Gemini 1.5 Flash workflow (63.63)
  • Latency: Gemini Flash 5.02–12.03 s vs. GPT-4o/o4-mini 15.70–18.43 s across workflows; verification roughly doubles latency for GPT-4o (18.08→28.16 s)
  • Token cost: DC 3-shot+Verification with o4-mini consumes 7.85 PT + 1.98 CT (k tokens) vs. baseline 4.60 PT + 0.74 CT

Limitations

  • Evaluation confined to BIRD Mini-Dev (500 samples); findings may not generalize to full BIRD Dev or other Text2SQL benchmarks
  • Only lightweight, industry-oriented workflows tested; upper performance bounds from more sophisticated pipelines (e.g., CHASE-SQL, MAG-SQL) are not assessed
  • Latency and token metrics are confounded by network conditions, server load, and model region—reported as indicative only
  • No fine-tuned or domain-adapted models included; all models used off-the-shelf
  • Error categorization in Figure 3 is model-assisted (o4-mini as judge), introducing potential classification noise

Relevance to Harnesses / Meta-Harnesses

This paper is a direct empirical study of agentic workflow harnesses: it defines a compositional notation for harness components (SQL Writer, Executor, Refiner, Feedback Provider, Candidate Selector) and benchmarks how stacking those components affects end-to-end system behavior. The finding that adding harness stages does not monotonically improve quality—and that retrieval preprocessing can actively harm accuracy—is a concrete warning for meta-harness designers who wire together tool-calling steps without domain-specific validation. The parallel majority-vote pattern (×5 with fallback Candidate Selector) is a reusable meta-harness primitive for answer aggregation, directly analogous to ensemble stages in broader agentic orchestration frameworks.