Skip to content

Scaling Generalist Data-Analytic Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Scalable data-analytic agents on open-source models; reduces proprietary model dependence

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DataMind is a scalable data synthesis and training recipe for building open-source generalist data-analytic agents that operate via code generation over diverse file formats. Trained on the curated DataMind-12K trajectory dataset, DataMind-14B achieves 71.16% average score across three benchmarks, outperforming GPT-5 and DeepSeek-V3.1. DataMind-7B leads all open-source models at 68.10%.

Problem

Existing data-analytic agents depend on proprietary models via prompt engineering or rigid multi-agent scaffolds. Open-source trained alternatives handle only compact tables and collapse on diverse-format, large-scale files (CSV, XLSX, SQLite) requiring long-horizon, multi-step reasoning. Three root causes: (1) no large-scale annotated training trajectories for data analysis, (2) unclear SFT/RL balancing strategy for long-horizon agent training, and (3) unstable multi-turn rollout under concurrent file I/O and limited memory.

Method

DataMind is a four-component pipeline:

Data synthesis. Files are collected from Kaggle (3,400 CSV, 560 XLSX) and BIRD/OmniSQL (1,954 SQLite). Queries are synthesized by feeding file metadata to DeepSeek-V3 under a 18-category fine-grained taxonomy. A recursive easy-to-hard composition chains 2–5 task types to produce multi-hop queries.

Trajectory sampling. A knowledge-augmented strategy provides category-level procedural workflow \(k\). \(N=3\) independent trajectories \(\{y^i\}_{i=1}^N \sim \pi_{\theta_\text{expert}}(\cdot|k,x)\) are sampled from DeepSeek-V3.1. A GPT-4o-mini judge checks self-consistency; inconsistent trajectories trigger a reflection loop \(\{y^i_\text{reflected}\}_{i=1}^N \sim \pi_{\theta_\text{expert}}(\cdot|k,x,\{y^i\},c)\). Three rule-based filters (format compliance, length ≤ 1024 tokens, linguistic integrity) yield 11,707 training instances (DataMind-12K).

Training. The model is jointly optimized with: $\(\mathcal{L}_\text{Final}(\theta) = \gamma \mathcal{L}_\text{SFT}(\theta) + (1-\gamma)\mathcal{L}_\text{DAPO}(\theta)\)$ where \(\gamma\) is cosine-annealed from 0.9 to 0.05. SFT masks environment tokens; DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization) filters trajectories with uniformly 0 or 1 rewards. Void turns (no valid code or answer) are masked entirely to prevent trajectory collapse.

Rollout stability. Asynchronized model generation and code execution decouple GPU/CPU peaks; chunked notebook-style code maintenance (retaining only text chunks, concatenating at runtime) reduces peak memory; per-trajectory sandboxed environments enforce CPU/memory caps.

Reward. Three components: binary format reward \(r_\text{format}\), model-as-judge answer reward \(r_\text{answer}\), and length penalty \(r_\text{length}\) that decays linearly from 1.0 to 0.5 between \(l_\text{min}=256\) and \(l_\text{max}=1024\) tokens, then applies \(-0.1\) beyond \(l_\text{max}\).

Key Contributions

  • DataMind pipeline: recursive easy-to-hard query synthesis + self-consistency-filtered trajectory sampling with a reflection rescue loop
  • DataMind-12K: 11,707 high-quality multi-turn ReAct trajectories spanning 18 task categories and three file formats
  • Dynamic \(\gamma\) scheduling that blends SFT exploitation and RL exploration in a single training run
  • Asynchronous + chunked rollout framework for memory-frugal, stable multi-turn code-based agent training
  • DataMind-7B/14B: state-of-the-art open-source data-analytic agents that surpass proprietary models

Results

  • DataMind-14B: 71.16% average (pass@1) across DABench, TableBench, BIRD — surpasses GPT-5 (69.44%), DeepSeek-V3.1 (70.58%), and all open-source models
  • DataMind-7B: 68.10% average (pass@1) — best among open-source 7B models; vs. Qwen2.5-Coder-7B baseline (ReAct) at 11.26%
  • On DABench pass@1: DataMind-14B 80.29% vs. DeepSeek-V3.1 81.32% (near parity), vs. Qwen2.5-Coder-14B baseline 71.21%
  • On BIRD pass@1: DataMind-14B 62.23%, DataMind-7B 59.41%; OmniSQL-7B reaches 57.11% on BIRD but drops to ~40% on TableBench/DABench, demonstrating DataMind's generalization
  • TableLLM/Table-R1 (trained on 20K instances) fail on DABench large-scale tables and BIRD multi-table analysis; OmniSQL/SQL-R1 (trained on 2.5M instances) still underperform DataMind trained on only 12K trajectories
  • Data scaling ablation: performance improves consistently from 2K to 12K training instances

Limitations

  • DataMind-12K is limited to 11,707 trajectories; coverage of task types may not fully represent all real-world analytic scenarios
  • Self-consistency filtering biases toward easier queries with more consistent answers; harder queries are partially rescued by reflection but may still be underrepresented
  • Answer reward relies on GPT-4o-mini as judge, introducing dependency on a proprietary model for both data construction and evaluation
  • Maximum 10 interaction rounds and file-size restrictions (20–1,000 rows) may exclude very large-scale or streaming datasets
  • RL training stability (void turn filtering, dynamic \(\gamma\)) required substantial empirical tuning; insights may not transfer directly to other domains

Relevance to Agentic AI / LLM Agents

DataMind directly advances the open-source frontier for code-executing LLM agents operating in multi-turn, tool-augmented environments — a canonical setting in agentic AI research. The dynamic SFT+RL training recipe and void-turn masking address known instabilities in long-horizon agent training, providing actionable techniques transferable to other agentic domains (e.g., software engineering, scientific experiment execution). The recursive easy-to-hard query synthesis and self-consistency trajectory filtering offer a general blueprint for constructing high-quality agentic training data where ground-truth trajectories are expensive to annotate. Surpassing frontier proprietary models with a 14B parameter open-source agent on complex multi-step reasoning benchmarks marks a significant step toward capable, accessible agentic systems for scientific discovery.