Skip to content

BioSecBench-Surveillance: A Verifiable Benchmark for AI Agents in Pathogen Genomic Surveillance

🕒 Published (v1): 2026-07-21 16:33 UTC · Source: Arxiv · link

Why this paper was selected

Verifiable benchmark for AI agents in pathogen surveillance; medical AI + agent eval combined

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

BioSecBench-Surveillance is a 100-task execution-based benchmark evaluating whether AI agents can autonomously select and run the correct genomic surveillance analysis pipeline given only raw sequencing data and task context. Across 16 model–harness configurations, the best pairing (Opus 4.8 / PI) achieves only ~50% pass rate, with failure dominated by wrong choices of references, thresholds, and normalization rather than wrong tool selection. The benchmark fills a gap: no prior execution-graded benchmark covered pathogen surveillance workflows.

Problem

Pathogen genomic surveillance is data-rich but bottlenecked on expert analysis: an analyst must choose among references, databases, filters, alignment/assembly strategies, lineage systems, and thresholds, all conditioned on organism, assay, and sampling context. It is unknown whether AI agents can reliably make these choices. Existing biology benchmarks cover epigenomics and pharmacology but none target the core tasks of pathogen surveillance; the closest work only tests sequence database retrieval, not analytical decision-making.

Method

Each evaluation snapshots a real surveillance workflow at the decision point, providing the agent with raw or near-raw sequencing data (FASTQ reads, assembled contigs, or aligned reads) stripped of all identifying metadata, plus a task prompt specifying the required answer schema. The agent must infer the correct pipeline, run it within a containerized sandbox (6 CPU cores, 34 GiB RAM, 512 GiB disk, 6-hour limit) preloaded with standard bioinformatics tools, and return a JSON-structured answer. Grading is fully deterministic: numeric fields are checked against relative/absolute/min/max tolerances, set-valued fields by Jaccard overlap above a threshold, and categorical fields by exact or normalized match. Ground truth is established with gold-standard validated workflows. Each evaluation is internally peer-reviewed. Sixteen model–harness configurations—pairing Anthropic (Opus 4.8/4.7/4.6, Sonnet 4.6), OpenAI (GPT-5.5/5.4), Google (Gemini 3.5 Flash, 3.1 Pro), and xAI (Grok 4.3, 4.20 Reasoning) models against Claude Code, PI, and Codex scaffolds—each ran 3 trials per evaluation (~4,800 total runs).

Key Contributions

  • A 100-task verifiable benchmark spanning 7 surveillance task categories (variant detection, taxonomic classification, AMR characterization, toxin/virulence characterization, genetic-engineering detection, source tracking, anomaly detection) across 6 sample types and both short- and long-read sequencing.
  • Deterministic, execution-based grading that scores the correctness of analytical choices, not just tool invocation.
  • Empirical evaluation of 16 model–harness configurations (3,962 gradable attempts), establishing a public performance baseline.
  • Identification of a consistent failure signature: agents select correct tools but misconfigure references, thresholds, and normalization.
  • A design requirement for benchmarks of this kind: all identifying metadata must be stripped from inputs to prevent agents from short-circuiting analysis via filename/header lookup.

Results

  • Best overall pass rate: Opus 4.8 / PI and GPT-5.5 / Codex tied at 50.2% (95% CI 40.1–60.3 and 40.8–59.6 respectively); lowest: Grok 4.20 Reasoning / PI at ~14%.
  • Average pass rate across all 16 configurations: 41%.
  • Refusal rates ranged from 0% (xAI) to ~31% (some Anthropic/OpenAI configurations under PI); refusals are excluded from the pass-rate denominator.
  • Hardest task category: anomaly detection (20% average pass rate); easiest: source tracking (50%).
  • Long-read datasets were substantially harder (26%) than short-read (41%); nucleic-acid target (DNA 39% vs. RNA 42%) and assay type (shotgun vs. targeted, both ~40%) had minimal effect.
  • Category difficulty ranking was largely conserved across all 16 configurations, indicating a structural difficulty signal rather than model-specific weakness.

Limitations

  • Tasks are unevenly distributed across categories, sample types, and technologies; aggregate scores are weighted toward better-populated labels.
  • Refusal exclusion from the denominator allows a gaming strategy (refuse hard tasks to inflate pass rate); no tested model exhibited this, but it is an unprotected attack surface.
  • Benchmark is restricted to tasks with a single objective ground-truth answer amenable to deterministic grading; open-ended judgment calls that characterize real surveillance (no agreed-upon method or threshold) are not covered.
  • Only a public subset is released; the full evaluation set is under restricted access for biosecurity reasons, limiting independent reproducibility.

Relevance to Agentic AI / LLM Agents

This benchmark directly probes a high-stakes agentic capability: multi-step scientific workflow execution where the agent must autonomously infer both the analytical pipeline and all its parameters from context. The finding that agents consistently select correct tools but fail on surrounding configuration choices (reference, threshold, normalization) is a generalizable diagnostic for agentic systems in expert domains—it distinguishes tool-selection capability from parameter-judgment capability. The execution-based grading methodology (inspired by SWE-bench) and the demonstration that the same model behaves very differently across harnesses (Claude Code vs. PI vs. Codex) are directly actionable for researchers designing agentic scaffolds and evaluation frameworks. For the LLM-agents field, this establishes biosurveillance as a new, adversarially-cleaned, and domain-authenticated benchmark suite.