Skip to content

StructureClaw: Traceable LLM Agents and an Executable Benchmark for Structural Engineering Workflows

🕒 Published (v1): 2026-07-16 12:13 UTC · Source: Arxiv · link

Why this paper was selected

Traceable LLM agents + executable benchmark for multi-step engineering workflows

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

StructureClaw is an artifact-centered workbench that routes LLM agents through governed engineering skills, typed tools, and shared artifact state to produce traceable structural-engineering workflows. It introduces StructureClaw-Bench, a 150-scenario executable benchmark requiring every intermediate artifact and execution assertion to pass. The automatic workflow raises mean Success Rate from 56.8% to 88.6% over a generic-skill baseline across 10 model configurations.

Problem

Existing evaluations of LLM systems for structural engineering measure final text outputs or generated scripts, not the complete evidence chain—interpreted requirements, structural model, validation records, solver output, code-check results, and report. A fluent or syntactically valid response can mask an incomplete, internally inconsistent, or non-executable workflow. No prior benchmark jointly requires structural model validity, backend-aware execution, safe non-execution, and trace-grounded reporting in a single evaluation environment.

Method

StructureClaw implements a ReAct-style execution loop over a shared structural-model protocol (a typed artifact carrying geometry, topology, materials, loads, and unit conventions). The agent selects from governed skills (each declaring supported structure types, validation requirements, backend constraints, and artifact contracts) and invokes typed tools (parameter extraction, model construction, validation, analysis, code checking, report generation) bound to backend providers (OpenSees/OpenSeesPy by default; PKPM/YJK when configured). Tool outputs update a persistent artifact state; downstream tools consume this state rather than reconstructing context from conversation history. Validation enforces schema compliance, cross-reference consistency, coordinate finiteness, and load-combination references before execution; failures trigger clarification, artifact repair, or explicit safe non-execution rather than prose substitution. The final report is generated only when the required execution record is present in artifact state.

StructureClaw-Bench contains 150 scenarios across three families: - Standard workflow (50): well-specified text-to-analysis requests across 8 structure types (steel frame, concrete frame, beam, truss, portal frame, continuous beam, column, generic). - Interactive robustness (50): missing/invalid/unrealistic/conflicting inputs, unit ambiguity, multi-turn updates, recovery, unsupported requests, static-analysis edge cases. - Multimodal reconstruction (50): geometry and topology inference from DXF files (30%) or images (70%) before model construction.

Success Rate is: $\(\text{Success Rate}(S) = \frac{100\%}{|S|} \sum_{s \in S} \prod_{a \in A_s} z_{s,a}, \quad z_{s,a} \in \{0,1\}\)$ No partial credit; every required assertion must pass in a single run.

Key Contributions

  • Artifact-centered formulation: defines agent success over a complete, mutually consistent evidence chain (intent → model → validation → solver → code check → report), making failures localizable to a specific stage.
  • Governed executable workbench: separates skills (domain planning context + contracts), tools (bounded state transitions), providers (backend bindings), and artifacts (typed shared state), enabling flexible yet auditable workflows.
  • StructureClaw-Bench: 150 executable scenarios balanced 50/50 Chinese/English, with typed assertions, reference fixtures, and multimodal inputs covering all three task families.

Results

  • Mean Success Rate across 10 configurations: 88.6% (automatic) vs. 56.8% (generic-only), a gain of +31.8 pp.
  • All 10 configurations improve in automatic mode; gains range from +24 to +46 pp.
  • Kimi-K2.6: 100/100 on standard cases (50/50); DeepSeek-V4-Flash and GLM-5.2: 48/50 each.
  • Interactive robustness (automatic mode only): top configurations (Claude Opus 4.8, GLM-5-Turbo, Kimi-K2.6) reach 94% overall, but invalid-value handling is a consistent weak point—diamond markers in Figure 6 show lower rates on that subset.
  • Multimodal reconstruction: evaluated on 6 configurations; the released comparator checks model counts, coordinate spans, and approximate loads (not full topology/connectivity), and fixture-consistent reconstruction remains a prominent open challenge.

Limitations

  • Single-run protocol with automatic retries disabled; a timeout counts as failure, so variance is not characterized.
  • The multimodal comparator evaluates only approximate structural counts and coordinate spans, not complete topology, support equivalence, or member-by-member connectivity—reconstruction scores are thus underspecified relative to full engineering validity.
  • Reference fixtures define reproducible operational targets, not the full set of engineering-equivalent solutions; the benchmark is a diagnostic testbed, not construction certification.
  • Provider support for PKPM and YJK requires external configuration; only OpenSees is available open-source, limiting backend diversity in the public release.
  • Subset differences (per structure type, language, modality) are explicitly noted as descriptive rather than statistically powered due to small per-cell counts.
  • The Auto–Generic Lift conflates skill routing, structural priors, artifact expectations, and validation guidance; it is a system-level comparison, not a component ablation.

Relevance to Agentic AI / LLM Agents

StructureClaw directly advances the practice of process-aware, artifact-grounded agent evaluation—a recognized gap in the field where final-response metrics miss workflow-level failures. Its separation of skills, typed tools, providers, and persistent artifact state is a concrete instantiation of governed tool-augmented agency (building on ReAct and PAL) in a high-stakes domain where non-executable outputs have real consequences. The paper demonstrates that domain-specific skill routing yields substantial, measurable gains (+31.8 pp) over generic tool use, providing empirical grounding for claims about the value of structured agent scaffolding. For researchers tracking agentic benchmarks, StructureClaw-Bench's all-or-nothing assertion protocol and its distinction between safe non-execution vs. unsupported termination are design patterns applicable to other domains requiring verifiable multi-step reasoning chains.