Skip to content

DataSage: Multi-agent Collaboration for Insight Discovery with External Knowledge Retrieval, Multi-role Debating, and Multi-path Reasoning

🕒 Published (v1): 2025-11-18 09:54 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DataSage is a multi-agent framework for automated data insight discovery that chains four specialized modules—dataset description, retrieval-augmented knowledge generation (RAKG), multi-role question raising, and multi-path code generation—in an iterative QA loop. It addresses three concrete failure modes of prior single-agent approaches: missing domain knowledge, shallow question formulation, and error-prone code generation. On InsightBench it outperforms the best prior agent by +7.5% insight-level and +13.9% summary-level G-Eval scores.

Problem

Existing LLM-based data insight agents operate in a question-driven paradigm where a single model generates questions, code, and insights sequentially. This produces (1) domain-blind analysis when datasets require external knowledge (e.g., healthcare, retail regulations), (2) shallow or generic questions from a single analytical viewpoint, and (3) frequent code errors from lack of iterative validation, leading to incorrect or misleading insights.

Method

DataSage structures the pipeline into four modules executed in \(N_{\text{iter}}\) iterative QA cycles:

  1. Dataset Description Module — extracts metadata, statistics, sample rows, and heuristic diagnostics into a structured JSON context \(D\).
  2. RAKG Module — a judge agent decides if external retrieval is needed (\(\text{Judge}(D, G) \rightarrow \{\text{yes, no}\}\)); if yes, \(N_q = 3\) Google-ready queries are generated, executed via Serper API, and a knowledge generator synthesizes structured knowledge items \(K\).
  3. Question Raising Module — a role designer generates \(N_R = 3\) domain-specific analytical roles (e.g., behavioral analyst, anomaly detector); each role independently raises questions (divergent phase); a global judge selects the best subset \(Q_j^*\) based on novelty, alignment, diversity, and complementarity with prior history \(H\) (convergent phase).
  4. Insights Generation Module — a question rewriter grounds each question; three CoT strategies generate parallel candidate code: Divide-and-Conquer, Query Plan, and Negative Reasoning; a code selector picks the best candidate \(c_0\); iterative code/plot reviewers and a code fixer refine it up to \(N_{\text{fix}} = 5\) times; a multimodal LLM interpreter processes both textual and visual sandbox outputs; a final judge selects the most valid insight \(I\) across all refinement iterations.

New questions in each cycle are conditioned on the accumulated question-insight history \(H\), enabling progressive depth of analysis.

Key Contributions

  • Four-module multi-agent framework (Dataset Description → RAKG → Question Raising → Insights Generation) operating as an iterative QA loop.
  • RAKG: on-demand, search-gated external knowledge retrieval with structured synthesis, activated only when internal LLM knowledge is deemed insufficient.
  • Divergent-convergent multi-role debating for question generation, replacing single-model question formulation with a role-diverse panel followed by a global judge.
  • Multi-path code generation using three distinct CoT strategies (Divide-and-Conquer, Query Plan, Negative Reasoning) with automatic code and plot review/repair cycles.
  • First use of multimodal LLMs (MLLMs) for insight interpretation, combining visual and textual sandbox outputs.

Results

All evaluations on InsightBench (100 tabular datasets, Easy/Medium/Hard splits), GPT-4o backbone, G-Eval metric:

  • Insight-level: DataSage 0.3530 avg vs. AgentPoirot 0.3284 (best baseline); +7.5% avg improvement (+7.8% Easy, +5.5% Medium, +9.3% Hard).
  • Summary-level: DataSage 0.4059 avg vs. AgentPoirot 0.3565 (best baseline); +13.9% avg improvement (+16.5% Easy, +12.6% Medium, +12.7% Hard).
  • DataSage outperforms all baselines across both metrics and all three difficulty levels.
  • Gains are larger on Hard datasets and larger on Summary-level than Insight-level, without explicit summary optimization.
  • Plot quality (GPT-4o-judged on Relevance, Clarity, Annotation, Interpretability, 0–10 scale): DataSage substantially outperforms AgentPoirot; ablations confirm both Code Refinement and Multi-path Reasoning components contribute.

Limitations

  • Evaluation is limited to a single benchmark (InsightBench, 100 datasets); generalization to other tabular analytics settings is unverified.
  • The external retrieval component introduces latency and dependence on a live search API (Serper/Google), reducing reproducibility and adding cost.
  • All results use GPT-4o as the sole backbone; performance with other LLMs is untested.
  • Multi-path code generation triples inference cost at the code generation stage; the paper does not report wall-clock or token cost comparisons.
  • The judge and role designer agents are themselves LLMs whose quality is assumed but not ablated independently.

Relevance to Harnesses / Meta-Harnesses

DataSage is a canonical example of a task-specific meta-harness: it does not perform analysis itself but orchestrates a pipeline of specialized sub-agents (judge, query generator, role designer, code selector, code reviewer, plot reviewer, code fixer, interpreter, final judge) with explicit inter-module data contracts and iterative feedback loops. The divergent-convergent debating structure and the multi-path code generation with automated repair loops are directly analogous to the judge-panel and adversarial-verify patterns used in general-purpose meta-harnesses. The RAKG module's conditional activation (retrieve only when needed) is a concrete example of routing logic that governs resource use within a harness—relevant to designing efficient, cost-aware orchestration systems.