Skip to content

PHITSBench: an execution-scored benchmark for AI-assisted PHITS radiation-transport input generation using natural language

🕒 Published (v1): 2026-07-08 16:35 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PHITSBench is an execution-scored benchmark of 282 tasks for evaluating LLM-assisted generation of PHITS Monte Carlo radiation-transport input files, spanning parameter editing, syntax repair, and full simulation generation from natural-language descriptions. Evaluation uses a Composite Metric Score (CMS) that combines binary execution success with a physics-fidelity score against reference transport observables. Without domain knowledge, LLMs score 0% on full generation; a structured machine-readable knowledge catalog plus agentic execution raises this to 57–73%.

Problem

Existing scientific-computing benchmarks evaluate LLM-generated code primarily on syntactic correctness (e.g., CodeBLEU), which fails to detect silently incorrect physics. PHITS—a widely used Monte Carlo particle-transport code with a legacy Fortran-card-deck input language—is underrepresented in foundation-model pretraining corpora, making it an especially stringent testbed. There is no prior execution-grounded benchmark for AI-assisted radiation-transport simulation that jointly assesses syntactic validity and physical fidelity.

Method

Three task tracks are defined over 282 tasks derived from the official PHITS 3.34 example library: - Edit (\(n=65\)): modify an existing valid input file per a natural-language instruction. - Repair (\(n=131\)): fix a single deliberately injected fault given the system error message. - Reproduce (\(n=86\)): generate a complete PHITS input from a natural-language physics description only.

Tasks are evaluated by actually executing PHITS and comparing generated transport observables to reference solutions via the Composite Metric Score: $\(\text{CMS} = w_{EX} \cdot EX + w_{PF} \cdot PF\)$ where \(EX \in \{0,1\}\) is binary execution success and \(PF \in [0,1]\) aggregates normalized \(L_2\) distance, integral agreement, and peak-position agreement across all expected tally outputs. Track-specific weights are \((w_{EX}, w_{PF}) = (0.5, 0.5)\) for Reproduce, \((0.2, 0.8)\) for Edit, and \((0.3, 0.7)\) for Repair. Success threshold is CMS \(\geq 0.95\).

Five GPT-5.4 configurations are evaluated: 1. Baseline: zero-shot, no external knowledge. 2. +PDF: full PHITS user manual injected via the Responses API. 3. +Cat+PDF: manual plus a custom 11-document, ~69k-token machine-readable PHITS knowledge catalog (parser constraints, keyword dependencies, undocumented defaults) generated via LLM-assisted source-code analysis and expert review. 4. Codex CLI: general-purpose coding agent with unrestricted iterations, xhigh reasoning, access to local PHITS installation and the +Cat+PDF knowledge. 5. Custom Multi-Agent: three role-specialized agents—Composer (initial generation), Manager (ReAct-style orchestration loop), Repairer (execution-feedback-driven correction)—with a fixed six-tool set (compose_phits_input, static_check_phits, run_phits, summarize_failure, inspect_tally_outputs, check_tally_config), bounded at five repair iterations, and RAG access to the knowledge catalog. Implemented via OpenAI Agents SDK v0.8.4.

Key Contributions

  • PHITSBench: first execution-grounded benchmark for PHITS input generation, with 282 expert-verified tasks across three workflow categories.
  • Composite Metric Score combining binary execution success and multi-component physics fidelity, with track-specific weighting.
  • A structured, machine-readable PHITS knowledge catalog (~69k tokens) derived from source-code parser analysis and expert review, capturing undocumented behaviors not present in the user manual.
  • Systematic ablation isolating contributions of pretraining knowledge, documentation injection, structured catalog, and agentic execution on simulation quality.
  • A domain-specific multi-agent architecture (Composer/Manager/Repairer) with bounded iteration and retrieval-based knowledge access, achieving competitive performance at 6Ă— lower cost than unrestricted Codex CLI.

Results

  • Edit track: Baseline 62/65 (95%), +PDF 60/65 (92%), +Cat+PDF 62/65 (95%) — effectively saturated at zero-shot.
  • Repair track: Baseline 92/131 (70%), +PDF 110/131 (84%), +Cat+PDF 116/131 (89%) — +19 pp gain from knowledge injection.
  • Reproduce track: Baseline 0/86 (0%), +PDF 28/86 (33%), +Cat+PDF 49/86 (57%), Codex CLI 63/86 (73%), Custom Agent 57/86 (66%).
  • Inference cost per task: Baseline ~\(0.01, +Cat+PDF ~\)0.20, Codex CLI ~\(0.82, Custom Agent ~\)0.13.
  • Failure analysis: remaining errors dominated by incorrect observable selection and tally configuration, not syntax generation.
  • CMS weight sensitivity: perturbing track weights by ±0.10 does not change method rankings.

Limitations

  • All results from single non-deterministic runs; no confidence intervals or variance estimates reported.
  • Benchmark is PHITS-specific; generalizability to other MC codes (MCNP, GEANT4, FLUKA) is not demonstrated.
  • Custom Agent's bounded five-iteration cap may artificially constrain performance relative to unrestricted Codex CLI.
  • Knowledge catalog required significant expert labor to construct; its applicability to other simulation codes is not established.
  • Remaining Reproduce failures (27–34%) stem from semantic physics reasoning (observable selection), which current approaches do not fully address.

Relevance to Harnesses / Meta-Harnesses

PHITSBench instantiates two distinct harness patterns simultaneously: the benchmark harness (which drives a real simulator, parses its output files, and computes multi-component fidelity scores as an automated evaluation loop) and the agent execution harness (the Custom Multi-Agent system wraps PHITS with a ReAct-style orchestration loop, domain-specific static checkers, and tally validators as a structured tool set). The paper's core finding—that structured machine-readable knowledge catalogs and execution-grounded feedback loops are necessary complements to foundation model capability—directly validates the harness design philosophy: wrapping legacy scientific tools in scaffolding that exposes structured feedback is as important as model strength. The bounded-iteration, cost-controlled Custom Agent versus the uncapped Codex CLI comparison offers a concrete cost/performance trade-off analysis for harness designers choosing between open-ended and budget-constrained agentic loops.