Skip to content

SpatialBench: Can Agents Analyze Real-World Spatial Biology Data?

🕒 Published (v1): 2025-12-26 07:40 UTC · Source: Arxiv · Venue: NEURIPS 2024 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SpatialBench is a benchmark of 146 verifiable problems drawn from real spatial transcriptomics workflows, testing whether frontier LLM agents can extract biological insight from messy, real-world spatial datasets. Across seven task categories and five spatial platforms, all evaluated frontier models perform poorly (20–38% accuracy under a base harness), with harness design emerging as a comparably important variable to model choice.

Problem

Existing agent benchmarks focus on general coding or biological knowledge QA, leaving a gap: can agents manipulate and reason over high-dimensional, messy, domain-specific data objects (e.g., AnnData spatial transcriptomics files) to produce scientifically correct conclusions? No prior benchmark captured this "data-interaction + domain reasoning" regime.

Method

Each of 146 problems snapshots a real analysis state (AnnData object + associated files) immediately before a target step, paired with a natural-language scientific prompt and a deterministic grader (Jaccard similarity, Precision@K, numeric tolerance, multiple-choice, or cosine similarity). Problems were constructed with domain scientists and validated against adversarial shortcuts (agents prompted to answer from prior knowledge without touching data). Agents run in an interactive Python environment; all runs are tripled and scored with two-stage t-based confidence intervals. Multiple harnesses are compared: a base configuration, Claude Code, and a proprietary Latch harness, all wrapping the same base models.

Key Contributions

  • 146-problem benchmark spanning 5 spatial transcriptomics platforms (AtlasXomics, MERFISH, Seeker, Visium, Xenium) and 7 task categories (QC, normalization, dimensionality reduction, clustering, cell typing, differential expression, spatial analysis)
  • Deterministic, automatically verifiable graders with anti-shortcut design requiring empirical data interaction
  • Quantitative evidence that harness design (tools, prompts, control flow, execution environment) contributes as much to performance as model choice
  • Trajectory-level behavioral analysis revealing distinct failure modes per model family (instruction-following collapse, domain miscalibration, unproductive thrashing)

Results

  • Overall accuracy (base harness): Opus-4.5 38.4%, GPT-5.2 34.0%, Sonnet-4.5 28.3%, GPT-5.1 27.4%, Grok-4.1 24.7%, Grok-4 22.8%, Gemini-2.5-Pro 20.1%
  • Harness effect (Opus-4.5): Latch 61.7% > Claude Code 48.1% > base 38.4%; the 23.3-point uplift from base to Latch exceeds the Opus–Sonnet gap under Claude Code (3.0 points)
  • Task stratification: Best models reach 51–75% on dimensionality reduction and spatial analysis but only 10–22% on QC and near 0% on QC for weaker models
  • Platform variation: 15–20 percentage-point swings across platforms for the same model; Seeker uniformly lowest (19–31%)
  • Step efficiency: GPT models average 2.1–2.4 steps/task at $0.02–0.04; Grok variants average ~10 steps at similar or higher cost; all 119 step-limit exhaustions (100 steps) occurred in Grok runs, all resulting in failure
  • Domain calibration: Opus-4.5 applies spatially appropriate min_genes thresholds (median 10) while other models default to scRNA-seq values (median 100–200), explaining its higher QC pass rate (~25% vs. ~5%)

Limitations

  • Deterministic graders can be brittle; discretizing scientific judgment into pass/fail loses nuance and may penalize scientifically equivalent answers
  • Each problem is a single snapshotted step; compounding errors in end-to-end workflows and iterative revision (e.g., revisiting QC after poor clustering) are not captured
  • Latch harness results are available only for Opus-4.5, preventing full cross-model harness comparisons
  • Coverage is limited to five spatial platforms and five tissue types; generalization to other spatial modalities or disease contexts is unverified

Relevance to Agentic AI / LLM Agents

SpatialBench provides a concrete, reproducible testbed for the underexplored agent capability of interactive data analysis in domain-specific scientific contexts—distinct from code generation or factual QA. The central finding that harness components (tools, prompts, control flow, execution environment) can swing accuracy by 23 percentage points reframes agent evaluation: the full agent stack, not just the base model, is the unit of measurement. The trajectory analysis identifying behavioral failure modes (format-error thrashing in Grok, domain miscalibration across models, non-productive multi-step exploration) offers actionable diagnostics directly applicable to any agentic system design. This work connects to the SWE-bench lineage but shifts the challenge from code correctness to empirical scientific reasoning under real-world data messiness.