AEC-Bench: A Multimodal Benchmark for Agentic Systems in Architecture, Engineering, and Construction

🕒 Published (v1): 2026-03-31 03:10 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AEC-Bench is a 196-instance multimodal benchmark for evaluating coding-agent harnesses (Claude Code, Codex) on real-world architecture/engineering/construction coordination tasks spanning single-sheet, cross-sheet, and cross-document reasoning. The paper shows that general-purpose harnesses partially transfer to AEC but fail on visual-spatial grounding and exhaustive traversal, and that augmenting base harnesses with domain-specific parsing tools (Nomic Parse/Embeddings) yields large gains on retrieval-sensitive tasks but hurts or has no effect on geometry-dependent tasks.

Problem

Coding-agent harnesses (SWE-bench style) are designed for structured, verifiable software environments. AEC coordination tasks are highly multimodal — construction drawings embed spatial geometry, leader lines, callouts, and cross-sheet references that collapse when treated as text files. No benchmark existed to systematically characterize where these harnesses succeed or fail in real AEC workflows, or to measure the effect of domain-specific tool augmentation on harness performance.

Method

Tasks are formulated under the Harbor harness (Harbor Framework Team, 2026): each instance provides a natural-language instruction, a sandboxed environment with real public-sector construction PDFs, pre-installed PDF CLI tools, and an automated verifier grading a JSONL structured output. Agents are free to write code, create intermediate cache files, and use terminal commands.

Two evaluation conditions per harness: - H (base): Codex or Claude Code with standard Bash + PDF CLI utilities (pdftotext, pdftoppm). - H+: same harness augmented with Nomic Parse (structured text/layout/reference extraction) and Nomic Embeddings for retrieval.

Task taxonomy has three scope levels: 1. Intra-Sheet — single-page reasoning (detail review, callout accuracy) 2. Intra-Drawing — multi-sheet navigation within a drawing set (cross-reference resolution/tracing, sheet-index consistency) 3. Intra-Project — cross-document coordination across drawings, specs, and submittals

Benchmark construction uses a semi-automated pipeline: domain experts inject controlled defects (mismatched callout labels, broken cross-references, swapped spec values) into real drawings with pixel-level differencing to ensure visual consistency. Ground truth is verified by professional engineers/architects.

Key Contributions

• AEC-Bench: 196 instances across 9 task families in 3 context-scope categories, grounded in real construction coordination workflows with automated evaluation
• Controlled defect injection pipeline with text-level assertions and pixel-level differencing for benchmark construction
• Systematic harness ablation (H vs. H+) across Codex (GPT-5.2, GPT-5.4) and Claude Code (Opus 4.6, Sonnet 4.6) isolating tool augmentation effects
• Identification of three distinct failure regimes: retrieval bottleneck, visual-spatial grounding failure, and judgment-heavy subjective evaluation
• Open release of dataset, harnesses, and evaluation code under Apache 2 license

Results

All scores are mean reward (0–100, higher = better); H+ = base harness + Nomic tools.

• detail-technical-review (Intra-Sheet): H+ gains average +32.2 pp across models (e.g., Claude Opus H: 35.7 → H+: 78.6; GPT-5.4 H: 35.7 → H+: 71.4)
• spec-drawing-sync (Intra-Project): H+ gains average +20.8 pp
• drawing-navigation (Intra-Project): H+ gains average +18.75 pp
• note-callout-accuracy (Intra-Sheet): H+ decreases average −3.6 pp; visual-required instances (10 of 14) achieve only 5% mean reward across all models, while text-catchable instances achieve 100%
• cross-reference-resolution (Intra-Drawing): H+ decreases average −2.4 pp
• submittal-review (Intra-Project): best reward 23.1 across all model/setup combinations; consistent low performance reflects judgment-heavy failure mode
• Tool usage: 77% of all trajectories invoke pdftotext; Codex agents use Bash 100% of interactions; Claude agents: 53% Bash, 35% Read; Codex uses pdftoppm rasterization in 96% of runs vs. 32% for Claude, yet increased image access does not improve visual grounding

Limitations

• Small scale: 196 instances total, limited document diversity and discipline coverage; statistical robustness is constrained
• Task families may not reflect the full breadth of real-world AEC drawing variation
• Controlled/injected defects may not capture the full diversity of organic drawing errors
• Evaluation relies primarily on deterministic keyword/structure matching against JSONL output; cannot fully capture nuanced engineering judgment
• Single trial per instance (no multi-run averaging), which may introduce noise
• Benchmark does not evaluate intermediate reasoning steps or tool usage quality, only final output correctness

Relevance to Harnesses / Meta-Harnesses

AEC-Bench is fundamentally a harness evaluation benchmark: its primary object of study is not the foundation model in isolation but the complete execution harness — tool set, document representation, sandboxing, and output contract. The H vs. H+ ablation design is a controlled experiment in harness augmentation, directly measuring how adding domain-specific parsing tools to an otherwise unchanged harness shifts performance profiles. The finding that the same augmentation simultaneously improves retrieval-sensitive tasks (+32 pp) and degrades geometry-grounding tasks (−3.6 pp) is a concrete harness design lesson: tool additions are not uniformly beneficial, and harness designers must profile failure modes before augmenting. The Harbor harness used as the execution substrate is itself a reusable meta-harness layer (task format + sandbox + verifier interface) that generalizes across domains — making AEC-Bench directly relevant to anyone designing domain-adaptive harness architectures.