CausalDS: Benchmarking Causal Reasoning in Data-Science Agents¶

🕒 Published (v1): 2026-07-09 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

CausalDS is a benchmark for evaluating LLM-based data-science agents on causal reasoning, jointly testing symbolic causal inference (across all three of Pearl's rungs), data-science execution, uncertainty quantification, abstention on non-identifiable queries, and tool use. Each instance is a fully synthetic scene built from a hidden structural causal model (SCM) with generated tabular data and a natural-language story. Evaluation on contemporary agents shows that performance across these five axes does not collapse to a single underlying capability.

Problem¶

Existing benchmarks bifurcate into (a) symbolic causal reasoning benchmarks with no realistic data analysis, or (b) data-science/coding benchmarks with no principled causal data-generating structure. Benchmarks constructed from real or familiar datasets risk the "causal parrot" failure mode—models exploiting memorized causal knowledge rather than genuine structure-sensitive reasoning—confirmed by concurrent work showing that variable anonymization sharply degrades causal benchmark accuracy. No prior benchmark jointly exercises all five of: causal graph reasoning, statistical estimation from code, uncertainty quantification, epistemic abstention on non-identifiable queries, and agentic tool use.

Method¶

Scene generation pipeline (five stages): 1. Graph sampling: DAGs are drawn from canonical motifs (chain, fork, confounding, IV, etc.) and optionally enlarged via anchor-based grafting, attaching auxiliary motifs through a single shared node to build more complex structures while preserving narrative coherence. Graph topology distributions are calibrated against empirical histograms from real-world causal datasets. 2. SCM instantiation and data synthesis: Each node is assigned a structural equation \(V_i = f_i(\text{Pa}_i, U_i)\) under typed mechanism profiles drawn from registries mixing empirical (biochemical-equation/Boolean-rule-derived) and synthetic families. Interventional/counterfactual ground truth is computed via Monte Carlo. 3. Observation layer (optional): Selected conceptual variables \(Z_j\) are withheld from the public table and replaced by noisy measurement bundles \(W_j = (W_{j1}, \ldots, W_{jd_j})\), where continuous measurements are \(W_{jr} = h_{jr}(Z_j) + \epsilon_{jr}\) and binary measurements are \(W_{jr}|Z_j \sim \text{Bernoulli}(p_{jr}(Z_j))\). Bundles are screened by Fisher information to guarantee recoverability. This layer controls data-science difficulty without altering the conceptual SCM or identifiability labels. 4. Verbalization: Abstract nodes are mapped to domain variables (optionally seeded from CauseNet) via an LLM mapper–auditor loop that iteratively checks graph-faithfulness. An LLM then generates and iteratively verifies a natural-language story. Grafted graphs are mapped stage-wise with shared anchor nodes immutable. 5. Task derivation and packaging: Tasks spanning Rung 1–3 are derived from the shared scene. Public artifacts (story, schema, parquet data, calibration split) are separated from private artifacts (ground truth, hidden test labels). Agents are evaluated using mini-swe-agent in sandboxed Docker containers with bash tool access and no network.

Identifiability gating: Rung-2 effect/estimation tasks are gated on ATE identifiability (decided via DoWhy/do-calculus/ID algorithm); Rung-3 tasks are gated separately using the ID/IDC algorithms (via the y0 library). Non-identifiable task instances set the correct answer to abstain, scored as binary correctness.

Scoring: Deterministic per-task metrics (RMSE, ROC-AUC, Brier, log-loss, \(F_1\), interval score). Normalized relative error: \(\text{NRelErr}_i = \frac{\sqrt{n_i^{-1}\sum_j(\theta_{ij}-\hat\theta_{ij})^2}}{1+s_i}\). Three pooled quality categories (PassRate, Med. NRel. Err, Med. \(F_1\)-Loss) combine into the headline CausalDS score: \(\text{CausalDS} = w_p(1-\text{PassRate}) + w_r \text{SNR} + w_f \text{Med.}\ F_1\text{-Loss}\), with task-count-proportional weights.

Key Contributions¶

SCM-grounded synthetic scenes with empirically anchored composition (variable type, graph structure, identifiability regime, mechanism profile, observation complexity distributions matched to real-world data).
Graph-faithful free-form verbalization via a mapper–auditor loop with optional CauseNet seeding for the outcome node and one neighbor, verified to not change answers on strong models under verbalization-swap ablation.
Separate observation layer that independently controls data-science estimation difficulty without altering causal estimands or identifiability—distinct from proximal causal inference.
Broad task suite across all three rungs of Pearl's hierarchy, including prediction, association, collider phenomenon, causal graph recovery, identification, ATE/ETT estimation, mediation (NDE/NIE), bias diagnostics, and counterfactual effect estimation.
Abstention-aware deterministic evaluation: non-identifiable queries expect and reward abstention; hallucinating an answer on a non-identifiable problem is explicitly penalized.
A 953-scene benchmark, each with three observation model variants (clean, proxy, proxy_hard).

Results¶

The text is truncated before quantitative model results tables are presented. The paper reports only qualitatively at this stage: - Evaluated contemporary agents on a 953-scene CausalDS exam compositionally grounded in real-world corpora. - "Model performance dissociates along the five axes": models differ in content correctness, uncertainty quantification, abstention recognition, and tool-use efficiency—the axes do not collapse to a single latent capability. - No specific model names, accuracy numbers, or leaderboard rows are present in the provided text.

Limitations¶

Full quantitative results are not present in the provided preprint excerpt; evaluation conclusions are stated qualitatively.
Verbalization quality depends on the generation-time LLM (Kimi 2.5 in the reported results), introducing a non-deterministic quality floor in story generation.
The observation layer guarantees recoverability via Fisher information screening but does not guarantee uniform estimation difficulty; residual variance across scenes may confound difficulty calibration.
Tasks are restricted to observational releases; agents cannot perform active interventions (experimental causal discovery is addressed by concurrent work, not CausalDS).
CauseNet seeding injects recognizable real-world causal pairs; the verbalization-swap ablation shows stability only for "strong-enough models," leaving weaker models potentially exposed to the causal-parrot failure mode.
Benchmark corpus size (953 scenes) may limit statistical power for fine-grained per-family comparisons across the full task taxonomy.

Relevance to Agentic AI / LLM Agents¶

CausalDS directly targets the agentic LLM use-case—an agent that must read files, execute code in a sandboxed environment, and produce structured answers—making it one of the few benchmarks that simultaneously measures reasoning quality and tool-use efficiency on a causal task. It raises the bar beyond code-writing benchmarks by requiring models to recognize when the correct behavior is to abstain, operationalizing a key safety-relevant capability (epistemic humility under uncertainty). The finding that performance dissociates across the five axes (causal reasoning, estimation, UQ, abstention, tool use) is directly informative for research into which components of LLM agent pipelines need targeted improvement. The fully synthetic, deterministic, contamination-resistant design also provides a methodological template for future agentic benchmarks in domains where ground truth is otherwise hard to compute.