Skip to content

Automating High Energy Physics Data Analysis with LLM-Powered Agents

🕒 Published (v1): 2025-12-08 18:13 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper presents the first LLM-agent-driven framework for automating High Energy Physics (HEP) data analysis, using a supervisor–coder agent architecture integrated with the Snakemake workflow manager. The authors demonstrate and benchmark 20+ LLMs on a Higgs boson diphoton cross-section measurement pipeline, defining quantitative metrics for success rate, error distribution, API efficiency, and cost.

Problem

LLM use in HEP has been confined to event-level inference or domain-adapted retrieval systems; no prior work has embedded LLM agents into end-to-end, reproducible collider-analysis pipelines. The gap is a framework that combines agentic code generation and self-correction with deterministic workflow orchestration for complex, multi-stage scientific computing.

Method

A hybrid system pairs a supervisor–coder agent (two roles implemented as separate API calls with distinct system prompts) with Snakemake as the DAG-based workflow orchestrator. The supervisor decomposes the user's natural-language task, instructs the coder to generate Python scripts, and reviews execution output; on failure, error messages are fed back in a self-correction loop capped at 3 retries (7 API calls max). The five-stage pipeline covers: (1) ROOT file inspection, (2) ntuple conversion to NumPy, (3) preprocessing/event selection, (4) signal-background separation via TabPFN, and (5) statistical categorization optimizing Higgs signal significance. Evaluation uses four metrics: per-step success rate, error category distribution (7 classes, classified by an auxiliary LLM), agent work ratio (Total Tokens − User Tokens) / User Tokens, and estimated dollar cost per step.

Key Contributions

  • First proof-of-principle of an LLM-agent-driven, reproducible HEP data-analysis framework embedded in Snakemake.
  • Quantitative evaluation methodology: success rate, error taxonomy (7 categories), agent-work efficiency ratio, API-call count, and per-step dollar cost.
  • Cross-model benchmark spanning 20+ LLMs (GPT-5, Gemini 2.5, Claude family, Grok, Llama-4, Qwen-3, DeepSeek-R1, O3/O4 series, open-weight gpt-oss-120b).
  • LLM-assisted error classification pipeline using a dedicated model (gpt-oss-120b) with structured prompts for reproducible failure categorization.
  • Identification of characteristic failure signatures per model family, beyond aggregate success rates.

Results

  • Gemini 2.5 Pro (219 trials): step success rates 58±3% (data prep), 88±2% (S-B separation), 74±3% (categorization); over 98% of tokens generated autonomously rather than from user input.
  • Top performers across all 5 steps: GPT-5 (10/10 on steps 1,3,4,5; 10/10 overall on step 2 S-B), Gemini 2.5 Pro (100% on steps 3,4,5), Grok-3 (100% on steps 1,3,4).
  • Step 2 (ntuple conversion) was consistently hardest — many models scored ≤30% there (e.g., O3: 20%, Gemini 2.5 Flash: 70%).
  • Failure modes: data-prep dominated by "all data weights = 0" and "intermediate file not found"; categorization dominated by function-calling and semantic errors.
  • Agent work: GPT-5 Mini shows the highest burst (~104× user tokens) at data prep; most models settle to 10–20× in later steps.
  • Cost: preprocessing is universally the most expensive step; S-B separation is cheapest; gpt-oss-120b on institutional compute has near-zero marginal cost.
  • Only 9 of 20 models successfully completed every step at least once.

Limitations

  • Temperature set to 0 but top-p/top-k not fixed; thinking-oriented models internally adjust sampling, producing stochastic variation.
  • Most non-baseline models tested with only ~10 trials per step — insufficient for statistical inference; results are qualitative indicators only.
  • Self-correction loop is capped at 3 retries; multi-step planning across the full DAG is not yet implemented (Snakemake provides the structure, not the agent).
  • No end-to-end success metric; each step is evaluated independently, so a model that fails step 2 can still score on step 3.
  • Observed significance not reported (proof-of-principle only, not a physics result).
  • Open-weight model cost advantage depends on institutional HPC access (LBNL CBorg), not universally reproducible.

Relevance to Agentic AI / LLM Agents

This work is a direct case study of multi-agent architectures (supervisor–coder pattern) deployed in a high-stakes, tool-use-heavy scientific environment, providing rare real-world benchmarking data for agent reliability and failure modes beyond general coding tasks. The agent-work and API-call metrics offer a principled vocabulary for measuring agentic efficiency — distinct from pure accuracy — which is broadly useful for evaluating agent frameworks. The finding that high success rate does not imply low agent work (some models succeed via extensive self-repair, others via single-pass reasoning) has direct implications for designing cost-effective agentic pipelines. The LLM-as-judge error classification loop is itself an agentic pattern of independent interest.