Skip to content

CLOSER-Bench: Evaluating Budgeted Cross-Stage Design Closure for Hardware Agents

🕒 Published (v1): 2026-07-18 04:28 UTC · Source: Arxiv · link

Why this paper was selected

CLOSER-Bench: long-horizon hardware agent eval with delayed/heterogeneous feedback; novel benchmark design for harness eval

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CLOSER-Bench is a controlled benchmark for evaluating hardware agents on budgeted cross-stage design closure—the ability to navigate from specification through RTL to GDS while managing a limited EDA tool budget across abstraction boundaries. It introduces stage-paired tasks (spec→RTL, RTL→GDS, spec→GDS) derived from a single shared design to isolate cross-stage coordination failure from per-stage incompetence. A ten-task pilot reveals a sharp completion–closure gap: all agents repair a localized AXI bug, but only one frontier agent kills hidden faults in the matched verification-closure task.

Problem

Existing hardware-agent benchmarks evaluate individual pipeline stages (RTL generation, PPA optimization, physical implementation) in isolation, using incompatible oracles and task designs. This makes it impossible to determine whether an agent that succeeds at the front end and backend separately will correctly allocate tool budget and recovery actions when stage boundaries are removed—i.e., whether it can perform genuine closure rather than a collection of independent code-generation steps. Pass-at-k metrics further fail to capture the continuous, long-horizon nature of EDA workflows where tool feedback is delayed, heterogeneous, and only partially localizing.

Method

CLOSER-Bench derives three tasks from a single design specification \(s\) and hidden objective set \(\Omega\): - Task A (spec→RTL): agent edits RTL and basic constraints; oracle is the front-end projection of the full signoff. - Task B (RTL→GDS): agent starts from a functionally correct but physically weak canonical implementation \(x_B\); oracle is the full signoff. - Task C (spec→GDS): agent has access to all cross-stage edit surfaces; same oracle as B.

A trajectory wrapper (built on the Harbor agent harness) records every call to four tool classes—simulation, synthesis/STA, full PnR, and final verifier—plus wall-clock time, modified files, edit layer, and best-so-far reward. Validity is gated by \(G(x,\omega) = G_{\text{func}}G_{\text{lec}}G_{\text{gls}}G_{\text{sta}}G_{\text{route}}G_{\text{drc}}G_{\text{lvs}}\) (Eq. 1). Quality is scored as: $\(R(x,\omega) = G(x,\omega)\,\exp\!\left(-\sum_j w_j\,\ell_j(x,\omega)\right), \quad \ell_j = \max\!\left(0, \log\frac{m_j(x,\omega)}{m_j^*}\right)\)$ Anytime progress is measured as the normalized area under the best-so-far reward curve: $\(\text{AUC}_B = \frac{1}{B}\int_0^B r^*(c)\,dc\)$ Budget tiers (Low/Medium/High) cap simulation calls (40/100/uncapped), synthesis+STA calls (10/25/60), and full PnR calls (3/8/20), with wall-clock limits of 4/10/24 hours. The shared design is firdma, a streaming FIR accelerator with AXI-Lite control, DMA engines, SRAM macro, and FIFO buffering, targeting Sky130HD via Verilator, Yosys, OpenROAD, and KLayout. Hidden evaluation uses pristine rebuilds with varied AXI widths, bursts, backpressure configurations, PVT corners, and floorplan seeds.

Key Contributions

  • Stage-paired protocol: derives three tasks (A, B, C) from the same specification and hidden conditions, enabling controlled attribution of performance differences to stage-boundary effects rather than unrelated task content.
  • Unified trajectory and anytime metrics: \(\text{AUC}_B\), time-to-first-feasible, time-to-threshold, rollback precision, and public–private gap, all grounded in a cost-annotated trajectory from Harbor.
  • Open-source executable harness: Verilator + Yosys + OpenROAD + KLayout + Sky130HD, with pinned image (openroad/orfs:26Q3-52-gc90beac09), structured ORFS per-stage JSON consumed directly by the trajectory layer.
  • Shared AXI/DMA vertical slice (firdma): 7 SystemVerilog modules + SRAM macro wrapper, with 23,789 synthesized cells and a validated global-routing flow.
  • Ten-task pilot suite spanning RTL repair, mutation verification, coverage, PPA optimization, DSE, cross-model debugging, and security, validating the harness and exposing reward-design failure modes (target saturation, false-alarm pathology).

Results

  • Physical-flow feasibility (Phase-0 ORFS): PicoRV32 (27,590 instances) completes in 18.3 min (+5.44 ns WNS); Chameleon (492,693 instances, 6 macros) in 22.3 min (WNS −0.04 ns); all three designs reach zero route DRC violations.
  • AXI completion–closure contrast (matched pair, PULP AXI data-width converter):
  • Repair task: all three agents (Codex/GPT-5.6, OpenCode/Big-Pickle, OpenCode/DeepSeek-V4) score 1.000 (2/2 trials each).
  • Verification-closure task: Codex kills 17/17 hidden faults (1.000) in 10.6 min; Big-Pickle kills 1/17 (0.0588) in 28.7 min; DeepSeek false-alarms on golden RTL (0.000) in 26.7 min.
  • fmax task: Codex reaches 181.7 MHz against a 120 MHz full-score target (oracle: 99.0 MHz; Big-Pickle: 110.1 MHz)—exposing reward cap saturation.
  • Coverage task: Codex reaches 88.7% line coverage against an 85% cap—showing line coverage alone is insufficient to measure checker strength.
  • Vortex multifault: one valid Haiku run repairs 2/3 faults (graded score), demonstrating the benchmark's graded-objective design.
  • Full three-way A/B/C matrix at all budget tiers is deferred to a future release pending firdma DRC/LVS closure.

Limitations

  • Single design, single PDK: the stage-paired vertical slice uses one primary design (firdma) and Sky130HD; open-source flows do not represent commercial signoff.
  • Incomplete firdma signoff: at submission, the macro-based flow has only reached global routing; final DRC/LVS and multi-corner STA remain pending.
  • Unequal pilot coverage: many tasks have only one valid trial; several are saturated by the current frontier; the pilot is diagnostic, not a model ranking.
  • Manually injected faults: synthetic fault injection does not substitute for real historical bug clusters.
  • Token-count incomparability: token costs are not directly comparable across agent scaffolds; wall-clock measurements depend on host contention.
  • No repeated A/B/C matrix yet: RQ1 (stage boundaries vs. efficiency), RQ2 (recovery confusion matrices), and RQ3 (budget-scaling progress curves) await a frozen signoff oracle and repeated trials.

Relevance to Harnesses / Meta-Harnesses

CLOSER-Bench is directly built on the Harbor agent harness, which provides the isolated agent/workspace/verifier layout that makes the benchmark's controlled trajectory recording possible—this is a concrete instantiation of a meta-harness that wraps heterogeneous EDA tools and enforces budget accounting externally to the agent. The trajectory wrapper functions as a harness-level instrumentation layer, intercepting all tool calls and enforcing budget tiers without agent cooperation, which is a general pattern for meta-harness design. The paper's stage-paired protocol also illustrates how a meta-harness can compose multiple sub-oracles (RTL simulation → synthesis/LEC → gate simulation → STA → route → DRC/LVS) into a single observable reward signal, providing a worked example of multi-stage oracle composition. For researchers building harnesses over long-horizon, multi-tool agent loops, the anytime \(\text{AUC}_B\) metric and cross-stage rollback tracking are directly transferable design primitives.