Skip to content

A Multimodal Conversational Agent for Tabular Data Analysis

🕒 Published (v1): 2025-11-23 11:21 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Talk2Data is a multimodal conversational agent for tabular data analysis that routes user queries (voice or text) through either code generation or direct narration, executing generated Python in a sandboxed runtime. The system orchestrates Whisper ASR, Qwen2.5-Coder, a LangGraph-based router, sandboxed execution, and Coqui TTS within a single agentic loop. Across 48 tasks on three datasets, the 7B model configuration achieves 95.8% accuracy with model-only latency of 1.15–1.64 s.

Problem

Existing NLIs for tabular data are either text-only (Tableau Ask Data, Power BI Q&A), lack multi-turn memory, or treat voice as a one-shot front-end without guarded code execution. No prior system unifies dual-modality input/output, transparent sandboxed code execution, adaptive response routing, and conversational memory in a single open-source, self-hostable agent.

Method

The pipeline has four stages:

  1. Input/context assembly: Dataset schema (column names, types, ranges, categorical exemplars) and ASR-normalized user utterance are packed into a compact, stable context that travels through the entire turn.
  2. Decision/orchestration (LangGraph node): A decision prompt classifies each turn as code_generation or chat_response via few-shot JSON output; this single fork determines which downstream branch executes.
  3. Branch execution:
  4. Code path: A code-generation prompt (constrained to whitelisted columns, expression-based outputs) drives Qwen2.5-Coder; the snippet runs in a sandboxed runtime (whitelisted libraries only, no filesystem/network/OS access, timeout + resource quotas). Numeric results return as JSON; figures as Base64.
  5. Narration path: A TTS-oriented chat prompt produces brief, speakable text grounded in metadata and dialogue history, rendered via Coqui VITS.
  6. Memory: Last \(N\) turns of query/output history are injected into each new prompt for multi-turn coherence.

Key Contributions

  • End-to-end multimodal agent unifying voice/text input with visual, tabular, and spoken outputs, with multi-turn dialogue memory over tabular corpora.
  • Explicit, auditable routing layer (single LangGraph decision node) separating intent classification from realization, enabling adaptive modality switching within one dialogue.
  • Sandboxed execution environment with whitelisted libraries, timeouts, structured error surfacing, and Base64 figure capture for transparent, reproducible code provenance.
  • Empirical 48-task benchmark across five Qwen2.5-Coder sizes (1.5B–32B) characterizing accuracy–latency–cost trade-offs.

Results

  • 7B model: 95.8% overall (37/39 code, 9/9 chat), 1 routing mistake, 1 execution error; model-only latency 1.15–1.64 s.
  • 14B model: 97.9% overall (38/39 code, 9/9 chat), 0 routing mistakes, 1 execution error — best overall.
  • 32B-AWQ: 95.8% overall, matching 7B but with 2 execution errors (sandbox blocks on ambitious code).
  • 3B: 87.5% overall; 1.5B: 62.5% with 17 routing misclassifications.
  • 7B identified as best balance for interactive use (accuracy vs. latency vs. cost).

Limitations

  • Benchmark is small (48 tasks, 3 datasets, manually evaluated); no held-out test split or inter-annotator agreement reported.
  • Each task run once per model — no variance or confidence intervals.
  • Sandbox restrictions cause legitimate failures (blocked dynamic imports, external API calls).
  • Conversational memory is a fixed last-\(N\) window; no semantic compression tested.
  • ASR and execution time excluded from reported latency figures, understating real-world end-to-end latency.
  • English-only ASR evaluation; multilingual capability is claimed but not tested.
  • No user study; usability claims are design-argument rather than empirically validated.

Relevance to Harnesses / Meta-Harnesses

Talk2Data is a concrete single-domain harness instantiation: it wraps heterogeneous components (ASR, LLM, sandbox executor, TTS) behind a centralized orchestration loop with an explicit routing node — a microcosm of the design patterns meta-harnesses formalize at larger scale. The explicit decision–realization separation (classify intent once, then dispatch to stable specialized branches) mirrors the control-flow discipline that meta-harness frameworks enforce across multi-agent pipelines. The sandboxed execution with structured error propagation and the schema-grounded context-pack pattern are directly transferable as harness primitives. The ablation across model sizes also demonstrates empirically how the harness's routing accuracy bounds overall system performance, a dependency structure meta-harness designers must account for.