EnvTrace: Simulation-Based Semantic Evaluation of LLM Code via Execution Trace Alignment -- Demonstrated at Synchrotron Beamlines

🕒 Published (v1): 2025-11-13 04:52 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EnvTrace evaluates LLM-generated instrument-control code by running it inside a digital-twin simulator, capturing time-ordered execution traces of hardware state changes, and aligning those traces against ground-truth traces. Benchmarking 31+ LLMs on 136 synchrotron beamline tasks shows top closed-source models approach human-level functional correctness, a result that syntactic metrics like exact-match and CodeBLEU systematically miss.

Problem

Syntactic metrics (BLEU, CodeBLEU, exact match) penalise functionally equivalent but structurally different code, and unit-test pass@k frameworks cannot capture the stateful, real-time, temporally ordered dynamics of cyber-physical instrument control. A motor commanded to \(+10\) rather than \(-10\) passes syntactic checks but causes physical damage; a for-loop rewritten as a while-loop fails exact match but is semantically identical.

Method

A control-logic digital twin of the NSLS-II CMS (11-BM) beamline is built from Docker containers running EPICS IOCs that expose the same Process Variables (PVs) as real hardware with state tracking but without full physics simulation. Both ground-truth and LLM-generated code are injected into a sandboxed IPython session communicating with the simulated EPICS environment via Channel Access. An EPICS camonitor daemon records every PV update as a tuple \((\text{pv\_name}, \text{value}, \text{timestamp})\), forming an execution trace. The two traces are aligned with difflib.SequenceMatcher to identify matched, missing, and extra events. A weighted composite score aggregates three dimensions:

\[\text{full\_score} = \begin{cases} 0.6 \cdot \text{pv\_match\_rate} + 0.2 \cdot \text{timing\_score} + 0.2 \cdot \text{temp\_score} & \text{(temperature tasks)} \\ 0.8 \cdot \text{pv\_match\_rate} + 0.2 \cdot \text{timing\_score} & \text{(otherwise)} \end{cases}\]

pv_match_rate is the fraction of correctly sequenced PV changes with correct values; timing_score combines \(R^2\) of a timestamp linear regression, slope deviation from 1.0, and MAPE of inter-event intervals; temp_score uses MAE over the full temperature profile and final-value error, decayed exponentially. A stricter binary "EnvTrace accuracy" requires exact sequence match, temporal adherence, and continuous-process fidelity simultaneously.

Key Contributions

• EnvTrace framework: simulation-based semantic code evaluation via execution-trace alignment, decoupled from syntax.
• Control-logic beamline digital twin: reproducible Docker + EPICS environment that runs unmodified Bluesky experiment code.
• Multi-faceted scoring: separates state-sequence correctness, temporal dynamics, and continuous-process fidelity into interpretable sub-scores.
• LLM leaderboard: 31 open- and closed-source models benchmarked on 116 simple-flow and 20 complex-flow beamline tasks, with human baselines on complex tasks.
• Pre-flight validation: the same simulator enables error detection in experimental plans before consuming scarce beamtime.
• Caching layer: avoids redundant simulator re-execution across models and prompt repetitions.

Results

• Top-tier closed-source models (e.g. Claude Sonnet 4 with Thinking) achieve pv_match_rate of 100% and full_score of 1.000 on simple-flow tasks, approaching human-level performance.
• Qwen2.5-coder on the same prompt: pv_match_rate 50%, full_score 0.600 due to hallucinated method name (aligned() vs align()).
• Systematic gap between exact-match accuracy and EnvTrace full score across all 31 models: the green "added correctness" segment in Fig. 3 confirms that syntactic metrics undercount functional correctness.
• Benchmark covers 116 simple-flow cases (single/few-line commands) and 20 complex-flow cases (loops, conditional logic); human scientists provide reference solutions for complex tasks.
• Each model is run 3× per prompt; results are averaged; caching removes simulator-timing variance from repeated runs.

(Paper is truncated; aggregate leaderboard numbers for all 31 models are not fully quoted above.)

Limitations

• Digital twin omits full physics: no synthetic detector images, no precise thermodynamic simulation — control-logic fidelity only.
• Ground-truth ambiguity: natural language prompts admit multiple valid implementations; fewer than 5 GTs per example means a high-scoring but unlisted variant still deflates the score.
• Domain coupling: the current implementation is Bluesky/EPICS-specific; generalising to other cyber-physical domains requires rebuilding the simulator layer.
• Scoring weights (0.8/0.2 split, temperature thresholds) are hand-tuned for beamline priorities and may not transfer without re-calibration.
• Binary "EnvTrace accuracy" is acknowledged as overly conservative; it conflates intent-mismatch with genuine errors.

Relevance to Harnesses / Meta-Harnesses

EnvTrace is itself a structured evaluation harness: it orchestrates code injection, sandboxed simulation, trace capture, alignment, and multi-dimensional scoring as a reusable pipeline around an LLM agent's outputs. The digital twin functions as the harness's execution substrate — a pattern directly analogous to sandboxed tool environments in agentic harnesses. The caching and multi-run averaging layers are meta-harness concerns (reproducibility, statistical robustness) applied to LLM evaluation. For researchers building harnesses that invoke LLM agents in stateful or cyber-physical loops, EnvTrace demonstrates how to replace brittle unit-test oracles with trace-alignment oracles that tolerate implementation diversity while still enforcing functional equivalence.