Skip to content

scBench-Long: Verifiable Benchmarking of Long-Horizon Single-Cell Biology

🕒 Published (v1): 2026-06-25 03:21 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

scBench-Long is a benchmark of 21 long-horizon single-cell biology tasks requiring AI agents to recover specific biological conclusions from raw data without prescribed methods. Across 1,068 trajectories and 17 model–harness pairs, the strongest pair (Claude Opus 4.8 + Claude Code) passes only 25.4% of runs, demonstrating that frontier agents fail at realistic multi-step scientific reasoning even when they succeed at individual analysis sub-steps.

Problem

Existing AI-biology benchmarks test isolated knowledge or local analysis steps (tool execution, single workflow stages). No benchmark evaluates whether agents can chain multi-step single-cell workflows—integrating RNA, chromatin, immune repertoire, and metadata—into verified, graded scientific conclusions. Verifiable ground truth is especially hard to define because the same dataset can support multiple valid claims, and published results do not always reproduce independently.

Method

Tasks are built around five published single-cell study systems (melanoma CD8 T cells, CD8 RNA+ATAC multiome, human–monkey chimeric embryos, KRAS-driven lung tumor aging, lethal COVID-19 lung pathology). Each evaluation provides raw or near-raw data, calibrated experimental context, auxiliary resources (TCR/BCR repertoires, ortholog maps, ligand–receptor databases), and a compact scientific question. Agents must produce answers drawn from a controlled vocabulary; grading is deterministic pass/fail on the final structured answer. A companion rubric-judge system (four LLM judges scoring task-specific "chokepoints") provides dense trajectory diagnostics but does not replace the endpoint score. Tasks are retained only after independent reproduction, peer review, distractor hardening, and multi-model trajectory inspection.

Key Contributions

  • 21 verifiable long-horizon single-cell evaluations spanning five study systems and five assay modalities (scRNA-seq, scATAC-seq, TCR/BCR-seq, Smart-seq2, snRNA-seq).
  • Deterministic endpoint grading over controlled answer vocabularies, paired with rubric-based trajectory diagnostics at hidden chokepoints.
  • 1,068-trajectory benchmark matrix across 17 model–harness pairs, including frontier models (Claude Opus 4.6–4.8, GPT-5.4–5.5, Gemini 3.1–3.5, Grok, Kimi, GLM).
  • Characterization of four recurring failure modes: prior substitution, raw-abundance confounding, association–mechanism conflation, and single-modality analysis under multimodal tasks.
  • Empirical demonstration that harness choice independently affects pass rates for the same base model.

Results

  • Best overall: Claude Opus 4.8 + Claude Code: 16/63 runs (25.4%; Wilson 95% CI 16.3–37.3%); any-pass on 8/21 tasks, all-replicate pass on 2/21.
  • Second: Gemini 3.5 Flash + PI: 14/63 (22.2%), any-pass 8/21.
  • Third: Claude Opus 4.8 + PI: 13/63 (20.6%).
  • Most model–harness pairs solve only 1–10% of runs; Grok 4.3 + PI passes 1/63 (1.6%).
  • 5 of 21 evaluations have zero successful trajectories across all models; the easiest task (pseudotime-delay) reaches 51.0% (26/51 trajectories).
  • Rubric scores correlate with endpoint success (Pearson \(r = 0.29\), Spearman \(\rho = 0.30\), AUC = 0.77); passing trajectories average 75.9% rubric score vs. 58.7% for failing ones.
  • Four rubric judges agree closely (mean pairwise \(r = 0.90\); minimum \(r = 0.87\)).
  • Harness changes endpoint pass count in 3–6 of 21 evaluations for each paired model; GPT-5.4 shifts from 2/63 (Codex) to 6/63 (PI).

Limitations

  • Only 21 evaluations; low statistical power for fine-grained model ranking (overlapping Wilson CIs across most pairs).
  • Controlled answer surfaces may penalize valid but unanticipated analysis paths not anticipated by benchmark authors.
  • Tasks drawn from five specific biological domains—findings may not generalize to other single-cell application areas.
  • Rubric judges are LLM-based and imperfect (AUC 0.77; high rubric score does not guarantee endpoint pass).
  • Replication ceiling: tasks require the exact target conclusion to be reproducible from staged data, which may exclude some scientifically valid but data-ambiguous claims.
  • Three replicates per evaluation per model–harness pair is a modest sample for estimating run-level reliability.

Relevance to Agentic AI / LLM Agents

scBench-Long is a direct stress test of the agentic capability gap: agents that execute individual tools correctly still fail to compose multi-step reasoning into verified scientific conclusions, showing that tool-use competence does not equal long-horizon task competence. The benchmark's harness-effect finding—that the execution environment (Claude Code vs. PI vs. Codex) independently shifts pass rates for the same base model—is directly actionable for agent-system designers who must select or build scaffolds. The identified failure modes (prior substitution, abundance confounding, association–mechanism conflation, modality-dropping) are domain-specific manifestations of general agentic weaknesses: lack of evidence integration, overreliance on training-set priors, and shallow multi-modal reasoning. As a verifiable, reproducible long-horizon benchmark, scBench-Long offers a template for rigorous agent evaluation in any domain requiring sequential, evidence-chaining decisions.