Skip to content

An Agentic Framework for Autonomous Materials Computation

🕒 Published (v1): 2025-12-22 15:03 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper presents a domain-specialized LLM-driven agentic framework for autonomously executing first-principles materials computations (via VASP), embedding physics domain expertise into modular workflow components. It also introduces the first rigorous benchmark (80 tasks across 4 computation types) for evaluating such agents. Across six frontier LLMs, the agent dramatically improves both task completion rates and result accuracy compared to bare LLM baselines.

Problem

Standalone LLMs cannot reliably automate computational materials science workflows: they hallucinate VASP parameters, misunderstand tag interdependencies (e.g., IBRION/POTIM coupling), fail to maintain workflow context across multi-step calculations, and lack mechanisms to verify physical convergence. Prior agentic approaches in this domain were limited to a handful of illustrative examples with no systematic benchmark for robustness evaluation.

Method

The framework wraps a predefined workflow library — each workflow encoding a validated, multi-step computational procedure (structural relaxation, band structure, adsorption energy, transition state search) — around a set of modular components: ReadFile, WriteFile, Command, RegexExtractor, and GetLLMAnswer. Given a user request and structure files (POSCAR), the LLM selects the appropriate workflow, then iteratively populates hierarchical prompt templates (integrating domain background, current simulation state, and output format constraints) to generate INCAR parameters via GetLLMAnswer, writes them to disk, executes VASP via Command, and parses outputs. Domain expertise is encoded in the prompt templates and workflow structure rather than in fine-tuned weights.

Key Contributions

  • First verifiable, end-to-end agent system for automating complex first-principles VASP calculations with physically coherent multi-step workflows.
  • A new benchmark of 80 computational tasks across 4 types (SR: 40, BS: 24, AE: 10, TS: 6) with ground-truth DFT reference data, evaluation scripts, and structured scoring metrics (completion rate + accuracy).
  • Demonstration that the agent framework closes the open-source/proprietary LLM performance gap: open-source models reach >90% completion with the agent vs. a >20% deficit without it.
  • Systematic failure-mode taxonomy: incorrect tag initialization, poor tag interdependence understanding, workflow context drift across steps.

Results

  • GPT-4o: completion rate 66.46% → 97.92% (+31.46%); accuracy 45.74% → 73.07% (+27.33%).
  • All six LLMs (DeepSeek-V3, GPT-4o, Qwen3-32B, o4-mini, Gemini-2.5 Pro, Claude-3.7-Sonnet) show consistent gains in both completion rate and accuracy with the agent.
  • SR tasks: average accuracy >95% with agent.
  • BS and AE tasks: average accuracy >80% with agent.
  • TS tasks: completion rate improves substantially, but accuracy remains persistently low due to NEB interpolation sensitivity and LLM misinterpretation of TS-specific parameters.
  • Reasoning models (Gemini-2.5 Pro, Claude-3.7, o4-mini) outperform standard models in both settings, particularly on TS tasks requiring long-range planning.

Limitations

  • TS task accuracy remains low even with the agent framework, due to NEB interpolation instability, surface reconstruction errors, and the need for manual inspection at multiple convergence checkpoints.
  • Framework is validated only on VASP; generalization to other simulation codes (e.g., Quantum ESPRESSO, CP2K) is unverified.
  • Benchmark is small in absolute terms (80 tasks total; only 6 TS examples), limiting statistical power for per-task conclusions.
  • The workflow library requires expert pre-design; the agent cannot construct novel workflows outside the predefined library.
  • No error recovery or self-correction loop: failures propagate downstream unless the workflow is restarted.

Relevance to Harnesses / Meta-Harnesses

This paper is a concrete instantiation of a domain-specialized harness: a modular orchestration layer that wraps LLM calls within predefined, expert-curated workflows, manages file I/O, executes external tools (VASP), and enforces structured output via prompt templates — the same architectural pattern as general-purpose agent harnesses but hardened for a specific scientific domain. The hierarchical prompt template system and workflow-selection mechanism directly parallel meta-harness concerns: routing tasks to specialized sub-agents, maintaining inter-step state, and constraining LLM outputs to physically valid formats. The benchmark contribution is also relevant: it establishes a reproducible evaluation protocol for harness-style agents in scientific computing, analogous to how general harness frameworks need standardized task suites to measure orchestration reliability vs. raw LLM capability.