GENIUS: An Agentic AI Framework for Autonomous Design and Execution of Simulation Protocols

🕒 Published (v1): 2025-12-06 11:28 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GENIUS is a domain-specific agentic framework that translates free-form natural-language prompts into validated Quantum ESPRESSO (QE) DFT simulation input files. It combines a structured knowledge graph with a tiered LLM hierarchy supervised by a finite-state machine (FSM) that orchestrates generation, validation, and iterative error repair. On 295 real-world benchmarks, ~80% of simulations complete successfully, with 76% of failures autonomously repaired.

Problem

Setting up and debugging Density Functional Theory (DFT) simulations in codes like Quantum ESPRESSO requires deep software expertise that domain scientists (chemists, physicists) often lack, creating a "know-do gap" between mature computational tools and their practical adoption. Existing automation approaches rely on rigid, predefined parameter templates and cannot diagnose or repair execution failures without human intervention.

Method

GENIUS implements a three-tier architecture governed by a FSM:

1. Recommendation System: A smart knowledge graph (KG) with 247 nodes and 330 edges, built from QE pw.x documentation. Retrieval combines keyword matching and context-aware graph traversal over 162 conditions across 9 categories (calculation type, pseudopotential, k-point settings, etc.). Nodes are embedded with a hashing vectorizer; top 70% by cosine similarity are selected and evaluated per-parameter to populate a structured input template.

2. Protocol Generation System: An LLM hierarchy — two worker models (databricks/dbrx-instruct, meta-llama/llama-3.1-405b-instruct) and a referee (anthropic/claude-3.5-sonnet) — generates the QE input file from the template. The generated file is immediately validated by executing QE; a zero exit code and absent CRASH file indicate success.

3. Automated Error Handling (AEH): On failure, the CRASH file is parsed by an LLM to extract error keywords, which query the KG for relevant documentation. The KG-augmented context is fed back to the LLM for correction. Each model in the hierarchy is allotted a fixed retry budget; exhausting retries triggers escalation to the next (more capable) model. The FSM manages entry, transition, error-recovery, and terminal states throughout.

Prompt complexity is scored with a 10-feature rubric (0–4 = basic, 5–8 = standard, ≥9 = complex). Material geometries are automatically fetched and standardized from Materials Cloud (MC2D/MC3D).

Key Contributions

• GENIUS framework: the first end-to-end agentic harness for autonomous DFT simulation protocol generation and repair
• Smart QE knowledge graph: 247 nodes, 330 edges, 162 explicit/implicit conditions enabling context-aware parameter retrieval
• FSM-governed AEH with tiered LLM escalation, eliminating manual debugging loops
• Benchmark dataset of 295 human-authored prompts from domain researchers naïve to QE syntax
• REST API (FastAPI) + web dashboard with Server-Sent Events for real-time monitoring
• Inference cost reduction: ~2× cheaper than LLM-only baselines; near-zero hallucinations on validated outputs

Results

• ~80% overall success rate across 295 diverse benchmarks
• 76% of initially failed runs autonomously repaired by the AEH loop
• Success probability decays exponentially with prompt complexity, reaching a 7% floor for the most complex prompts
• Halves inference cost compared to LLM-only baselines
• Virtually eliminates hallucinations (relative to LLM-only) by grounding generation in KG-retrieved documentation

Limitations

• Validated exclusively on Quantum ESPRESSO pw.x; generalization to other atomistic codes (e.g., VASP, CP2K) is claimed but undemonstrated
• KG quality — especially edge conditions — required manual refinement and remains an acknowledged weak point; improvement depends on future community contributions
• AEH effectiveness is bounded by the informativeness of QE CRASH file messages; opaque errors are harder to resolve
• Prior failed-retry logs are discarded during AEH to save context, potentially losing useful diagnostic signal
• The 70% cosine similarity cutoff increases computational cost; optimal threshold is not systematically characterized
• Exponential success decay at high complexity implies a hard practical ceiling for highly underspecified or ambiguous prompts

Relevance to Harnesses / Meta-Harnesses

GENIUS is a textbook domain-specific meta-harness: the FSM acts as the orchestration backbone, explicitly managing which agent runs, when it retries, and when control escalates to a more capable model — the defining pattern of a supervised multi-agent harness. The three-stage pipeline (recommend → generate → repair) with typed state transitions and budget-limited retries mirrors meta-harness designs that separate the control plane (FSM, retry logic, escalation policy) from the execution plane (individual LLM calls). The KG-grounded retrieval step demonstrates how a harness can inject structured domain knowledge into agent prompts to suppress hallucination — a generalizable technique for any harness operating in a constrained formal-language domain. Researchers tracking harness architectures will find GENIUS a concrete, production-tested example of FSM-supervised LLM orchestration with automated failure recovery.