Skip to content

EngiAI: A Multi-Agent Framework and Benchmark Suite for LLM-Driven Engineering Design

🕒 Published (v1): 2026-05-19 12:12 UTC · Source: Arxiv · link

Why this paper was selected

EngiAI: multi-agent benchmark combining simulation, retrieval, and manufacturing preparation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EngiAI is a LangGraph-based multi-agent system with a supervisor-router architecture that coordinates seven specialized agents (engineering, RAG, ArXiv, search, HPC, CLI, 3D printer) for end-to-end engineering design. The paper simultaneously introduces a three-dimensional benchmark suite—workflow execution across seven cognitive-demand prompt styles, gated RAG evaluation, and HPC orchestration scoring—evaluated across four LLM backends. Proprietary models reach 96–97% task completion on the structural benchmark; open-source 4B models reach 55–78%, with conditional branching as the hardest failure mode.

Problem

Existing agent benchmarks (AgentBench, ACEBench, Ď„-bench, FDM-Bench) evaluate single-agent, single-call, or single-phase tasks and do not expose failure modes unique to multi-agent engineering pipelines: chaining numeric outputs across tool calls, maintaining numerical precision under competing plausible values, conditional branching on runtime outputs, and long-horizon HPC orchestration across SLURM clusters. No prior framework unifies topology optimization, RAG-grounded parameter selection, manufacturing export, and HPC training orchestration in a single evaluable system.

Method

EngiAI follows a hierarchical supervisor pattern implemented in LangGraph. A central supervisor LLM receives each user prompt, classifies intent, and routes to one of seven specialist agents (each a self-contained LangGraph state machine with a bounded tool set). The supervisor can chain agents sequentially and enforces termination on repeated re-routing. Tool decomposition is motivated by empirical findings that LLM tool-selection accuracy degrades with tool-set size, so each specialist keeps its effective tool list small.

The benchmark evaluates three complementary axes: 1. Workflow benchmark — seven prompt styles (FULL, NATURAL, W-RAND, W-DERIVED, W-DISTRACT, W-COND, W-MULTI) targeting direct tool use, semantic disambiguation, arithmetic derivation, conditional branching, and working-memory retention. The composite score is design quality (65%, six sub-metrics: IoU, pixel accuracy, objective/constraint match, 2D connectivity, 3D watertightness) + tool use (20%, correct-call ratio) + task completion (15%, binary with parameter tolerance ±0.05). 2. RAG benchmark — four handcrafted prompts with escalating retrieval demand evaluated under three conditions: RAG-on, RAG-off, and Empty-RAG (tools present but index empty). A gated scoring mechanism credits parameter accuracy only when search_documents is actually invoked, preventing parametric-memory confounding. 3. HPC benchmark — four-step SLURM pipeline (script generation → job submission → monitoring → model evaluation) scored as step completion (70%) + configuration correctness (15%) + metric extraction (15%), under EXPLICIT and NATURAL prompt formats.

Key Contributions

  • Seven-style workflow benchmark isolating cognitively distinct failure modes in multi-agent engineering pipelines, including novel W-DISTRACT and W-COND styles absent from prior benchmarks.
  • Gated RAG scoring that cleanly isolates document-retrieval contribution from parametric memorization, with an Empty-RAG ablation condition.
  • First HPC orchestration benchmark for LLM agents on real SLURM clusters, evaluating end-to-end ML training pipeline completion.
  • EngiAI reference implementation in LangGraph: supervisor + seven specialists integrating EngiBench/EngiOpt, MMORE retrieval, SLURM SSH, and a Prusa 3D printer MCP server.

Results

  • Workflow (Beams2D): GPT-5-mini 96% TC / 0.68 CO; Gemini-3-Flash 97% TC / 0.69 CO; Qwen3-4B 55% TC / 0.53 CO; Qwen3.5-4B 78% TC / 0.63 CO.
  • Conditional branching (W-COND) is hardest: TC drops to 20–53% on Photonics2D across all models (vs. 87–93% on Beams2D for proprietary models); Qwen3-4B reaches only 20% on Photonics2D W-COND.
  • Domain transfer (Photonics2D): average TC falls to 73–84% across all models, roughly 10–15 pp below Beams2D equivalents.
  • RAG gating: RAG-on yields near-perfect scores (~1.0); RAG-off yields near-zero, validating that the benchmark isolates retrieval contribution rather than parametric knowledge.
  • HPC orchestration: one proprietary model completes all four SLURM pipeline steps in 100% of runs; another drops to 50%, confirming that multi-step instruction following degrades in long-horizon workflows.
  • Clear generational improvement within the open-source 4B tier: Qwen3.5-4B outperforms Qwen3-4B by ~23 pp TC on Beams2D average.

Limitations

  • Evaluated on only two EngiBench problems (Beams2D, Photonics2D); generalization to other physics domains (fluid, thermal, electromagnetic) is untested.
  • 3D watertightness sub-score is structurally deflated for all models due to threshold-based STL extraction producing non-manifold meshes—this reflects a pipeline artifact, not agent capability.
  • Open-source model evaluation is limited to 4B-parameter models served via Ollama; mid-size (7B–70B) open-source models are not included.
  • Residual run-to-run variance despite temperature=0 and fixed seed (JSON parsing errors dropped 1–2 runs in some Qwen3.5-4B cells, leaving \(n=13\)–\(14\)).
  • HPC benchmark uses only the cGAN algorithm; diffusion model results are relegated to appendix and the NATURAL-format HPC results are not fully detailed in the main text.
  • Source code was not yet released at submission time.

Relevance to Harnesses / Meta-Harnesses

EngiAI is a direct instance of a domain-specific meta-harness: a supervisor agent that dynamically routes subtasks to specialized sub-agents, each with its own tool set and state machine—precisely the orchestration topology that harness researchers study. The paper operationalizes the standard harness design heuristic that decomposing tool space across specialists improves selection accuracy, and empirically confirms it degrades gracefully when individual agents chain results. The three-dimensional benchmark provides a reusable evaluation scaffold for comparing harness architectures—particularly the gated RAG scoring and the W-COND/W-MULTI styles, which measure properties (conditional dispatch, cross-agent state propagation) central to meta-harness correctness. The HPC benchmark further demonstrates harness evaluation in long-horizon, async-tool settings (SLURM), extending beyond the synchronous API-call regimes most benchmarks assume.