BIMgent: Towards Autonomous Building Modeling via Computer-use Agents

🕒 Published (v1): 2025-06-08 16:45 UTC · Source: Arxiv · Venue: ICML 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

BIMgent is a multi-agent agentic framework that autonomously authors 3D Building Information Modeling (BIM) files by operating BIM authoring software (Vectorworks) through GUI computer-use — perceiving screenshots, planning hierarchically, and executing keyboard/mouse actions. It introduces a three-layer pipeline (Design → Action Planning → Execution) with RAG-driven low-level planning, dynamic GUI grounding, and LLM-as-judge supervision. On a custom 25-task benchmark, BIMgent achieves 32% end-to-end success versus 0% for GPT-4o and Claude 3.7 baselines.

Problem

Computer-use agents for general desktop automation do not generalize to highly specialized, long-horizon domain tasks. BIM authoring involves: (1) translating open-ended conceptual design into precise software commands; (2) navigating complex parameterized GUIs with many interaction modes; (3) filtering visually noisy interfaces for relevant UI elements; and (4) maintaining correctness across ~100 sequential interdependent actions where a single error causes cascading failures.

Method

BIMgent is structured as three layers:

Design Layer: Converts text descriptions or 2D floorplan sketches into a coordinate-aligned 2D floorplan. A GAN or multimodal LLM generates the floorplan image; DeepFloorPlan segments walls/openings; a VLM proofreads and extracts fine-grained element metadata; a rule-based mapper converts image coordinates \((x_i, y_i)\) to GUI pixel coordinates via \(x_{gui} = \frac{x_i}{w_{img}} \cdot w_{gui}\), \(y_{gui} = \frac{y_i}{h_{img}} \cdot h_{gui}\).

Action Planning Layer: A high-level planner (GPT-4.1) decomposes the workflow into element-level general steps following architect modeling patterns. A low-level planner uses RAG over official Vectorworks documentation (text-embedding-3-small + vector DB) to generate detailed substeps of two types: Pure-Action (deterministic keyboard shortcuts/placements, pre-generated as full sequences via speculative multi-action execution) and Vision-Driven (require GUI-state-conditioned grounding).

Execution Layer: Pure-Action substeps execute the pre-planned action batch; a supervisor (o4-mini) checks BIM metadata displayed by the software and triggers undo+regenerate on failure. Vision-Driven substeps use dynamic GUI grounding: diff the current screenshot against a step-start reference to isolate changed regions (typically pop-up windows), parse that region with OmniParser-v2 into Set-of-Marks, then feed a VLM action generator the grounded UI elements. After each substep, the supervisor re-evaluates GUI state and triggers self-reflection if needed.

Key Contributions

• BIMgent: first end-to-end agentic framework for autonomous 3D BIM authoring via GUI computer-use, handling both text-to-building and floorplan-to-building tasks.
• Hierarchical two-level planner with RAG over domain-specific software documentation, separating Pure-Action from Vision-Driven substeps.
• Dynamic GUI grounding via screenshot differencing to suppress visual noise and focus grounding on active pop-up regions.
• LLM-as-judge supervision loop with undo-and-regenerate for Pure-Action and self-reflection for Vision-Driven substeps.
• Mini Building Benchmark: 25 real-world BIM tasks (>2000 action steps) with human architect evaluation across design quality (6 criteria) and per-element operation success rates.

Results

• BIMgent end-to-end success: 32% vs. GPT-4o 0% and Claude 3.7 0% on Mini Building Benchmark (25 tasks).
• Element-level success rates — Walls: 86.58%, Openings: 95.12%, Slabs: 73.81%, Layers: 46.34%, Roof: 60.00% (vs. Claude 3.7 best sub-element: walls 31.70%, openings 35.36%).
• Ablation — removing dynamic GUI grounding drops end-to-end from 32% to 13.33% (−18.67 pp); removing supervision drops to 11.53% (−20.57 pp); removing hierarchical planning collapses to ~0% (N/A end-to-end, wall success 41.46%).
• Design quality: average score ≥3/5 across all six architect-evaluated criteria, outperforming Claude 3.7 SVG-based baseline on all criteria.

Limitations

• Benchmark is small (25 tasks, single software: Vectorworks); generalization to other BIM tools (Revit, ArchiCAD) is untested.
• 32% end-to-end success remains low for practical deployment; roof and layer creation are weak points.
• Human evaluation introduces inter-rater subjectivity; no automated task-completion oracle exists for BIM output.
• Relies on a heterogeneous model stack (gpt-image-1, Gemini 2.5 Pro, GPT-4.1, o4-mini) — cost and latency for 100+ action tasks not reported.
• No evaluation of recovery depth: how many retries the self-reflection loop consumes per failed substep.
• Domain-specific RAG requires pre-indexed vendor documentation; adaptation to new software demands fresh documentation ingestion.

Relevance to Harnesses / Meta-Harnesses

BIMgent is a concrete domain-specific meta-harness: it orchestrates a pipeline of heterogeneous specialist agents (floorplan generator, segmentation model, VLM interpreter, high-level planner, low-level RAG planner, action generator, supervisor/judge) each invoked at appropriate stages, with shared execution state (memory module storing trajectories for cross-component cooperation). The supervisor-as-judge pattern — issuing undo+retry or reflection prompts to downstream agents — is a direct instance of the harness-level control loop where a meta-controller governs sub-agent behavior rather than leaving recovery to individual agents. The speculative multi-action pre-generation (borrowed from UFO-2) and the two-workflow dispatch (Pure-Action vs. Vision-Driven) illustrate how a harness can route work to structurally different execution backends based on substep type, a generalization relevant to any harness targeting heterogeneous tool ecosystems.