Autonomous Pressure Control in MuVacAS via Deep Reinforcement Learning and Deep Learning Surrogate Models¶

🕒 Published (v1): 2025-12-17 15:19 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper presents a fully data-driven pipeline for autonomous pressure regulation in MuVacAS, a nuclear fusion materials testbed, by training a Fourier Neural Operator (FNO) as a digital-twin surrogate of the argon injection system and then using it as a simulation environment to train a PPO-based deep reinforcement learning agent. The trained agent was successfully deployed on the physical prototype, maintaining target pressures despite a distribution shift in injector geometry not seen during training. It outperforms PID controllers tuned under the original configuration.

Problem¶

Precise pressure regulation in ultra-high vacuum systems (like MuVacAS, the IFMIF-DONES accelerator testbed) is complicated by non-linear dynamics and fast, unpredictable disturbances. Traditional PID controllers require conservative tuning and fail to generalize across configuration changes. Direct DRL training on physical hardware is impractically slow (~1 second per control step in real hardware vs. conventional simulators) and risks damaging equipment.

Method¶

Two-stage data-driven approach:

Deep Learning Surrogate Model (DLSM): A Fourier Neural Operator (FNO) is trained on ~9 hours of real MuVacAS operational data (~30,000 environment transitions at 1 Hz) to learn the mapping (q_t, p_t(z)) → p_{t+1}(z), i.e., next-step spatial pressure distribution given current argon injection rate. The FNO operates in frequency space via FFT, capturing long-range spatial correlations along the beamline. Training used NVIDIA Modulus with 6 FNO layers, 12 Fourier modes, Adam optimizer.
DRL agent: A PPO agent (Stable-Baselines3, [64,64] tanh MLP policy) is trained inside a Gymnasium environment wrapping the FNO surrogate. The 3D observation space is {log-pressure at collision chamber, log target pressure, current injection rate}; action is a scalar injection rate. Reward is exp(-k|p_t - p*|) with k=4, encouraging tight pressure tracking with smooth gradients. Training ran for 270k steps (budget: 1M).

The trained policy was then deployed directly on the physical MuVacAS prototype with safety protocols (automatic shutdown on limit violation, human override capability).

Key Contributions¶

First deployment of a DLSM–DRL pipeline on a real particle accelerator prototype (not simulation-only)
FNO-based digital twin trained on real operational data enables orders-of-magnitude faster DRL training than physical or Monte Carlo (Molflow) simulators
Demonstrated zero-shot sim-to-real transfer under distribution shift: agent trained with injector at collision chamber, deployed with injector repositioned further upstream
Outperforms PID controllers that failed under the modified injection geometry

Results¶

Agent successfully reached and maintained all assigned pressure targets in the physical MuVacAS prototype across a sequence of progressively complex experiments (fixed targets and dynamic setpoints)
PID controllers tuned under the original configuration failed to achieve satisfactory performance under the modified injector position; DRL agent succeeded despite the same distribution shift
Asymmetric control effort observed: reaching lower pressures required more control actions than raising pressure, consistent with system physics
In short-interval tests (50s transition windows), the agent could not always reach target within the window due to injector placement; extending transition intervals resolved this

Limitations¶

Sim-to-real gap remains: modified injector geometry required more control actions than training conditions predicted; fine-tuning on real hardware not yet done
Gaussian random field training data does not capture static or slowly-varying injection states, reducing surrogate accuracy in those regimes
Observation space includes sensor inside the collision chamber; final accelerator design requires excluding that sensor, necessitating future redesign
No quantitative benchmark comparison against PID (qualitative only)
Short paper (NeurIPS 2025 workshop); ablation studies and offline RL / imitation learning comparisons are future work

Relevance to Agentic AI / LLM Agents¶

This paper is an instance of the surrogate-model-as-environment paradigm for agentic control: a learned world model (FNO digital twin) replaces expensive or dangerous real-world interaction during agent training, then the agent transfers to the real system. This pattern — train agent in a learned simulator, deploy in reality — is directly analogous to model-based RL approaches increasingly studied in LLM agent settings (e.g., learned environment models for tool-use or web agents). The demonstrated robustness to distribution shift (new injector geometry not seen in training) is relevant to agentic AI research on generalization and sim-to-real transfer. For researchers tracking agentic AI, this work is a strong applied proof-of-concept that data-driven digital twins can make DRL-based autonomous agents viable in high-stakes physical systems where direct online learning is infeasible.