Agentic Hardware Design as Repository-Level Code Evolution

🕒 Published (v1): 2026-06-26 17:21 UTC · Source: Arxiv · link

Why this paper was selected

NVIDIA (Khailany); hardware design as repo-level code evolution—novel agentic framework with executable evaluator

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HORIZON treats RTL hardware design as repository-level code evolution: a structured Markdown harness is compiled into a project pack (evaluator, acceptance predicate, git/runtime policy, domain skills), and a fully hands-free agent loop evolves an isolated git worktree until correctness gates are satisfied. Evaluated on ChipBench, RTLLM-2.0, Verilog-Eval-v2, and nine CVDP categories using GPT-5.3, it achieves 100% pass rate across all suites—at the cost of up to 82 iterations and ~210M tokens total (~91% cached).

Problem

Single-turn RTL code generation is insufficient: correct Verilog requires cycle-accurate temporal behavior, precise bit widths, reset/interface conventions, and FSM transitions that are frequently underspecified in natural language. Existing iterative/agentic RTL systems (AutoChip, RTLFixer, VerilogCoder, MAGE, ACE-RTL) are all built around per-module generation-and-repair pipelines and do not provide a general repository-managed substrate for systematic benchmark-level evaluation. Prior repository-scale self-evolution work (AlphaEvolve, SATLUTION, ABCEvo) evolved EDA software, not hardware design artifacts themselves.

Method

Harness compilation. A user-authored Markdown harness \(m\) is compiled by a bootstrap agent \(G_\phi\) into a project pack: $\(p = G_\phi(m),\quad p = (\pi_{\text{agent}},\, E_p,\, A_p,\, \Gamma_p,\, \Omega_p)\)$ where \(\pi_{\text{agent}}\) is the agent policy prompt, \(E_p\) is an executable evaluator (compile/simulate/coverage/assertions), \(A_p\) is the acceptance predicate, \(\Gamma_p\) is the git/runtime policy, and \(\Omega_p\) encodes domain skills.

Agent loop. At each outer step \(t\), the agent reads the current worktree state $\(s_t = (\text{tree}(w_t),\, p,\, z_t,\, \ell_{\leq t},\, \mu_t)\)$ and proposes an option \(a_t = (\Delta_t, u_{t,1:K_t}, \rho_t)\) (patch, tool calls, review decision). The evaluator produces evidence \(y_t = E_p(w_t \oplus \Delta_t)\), and the acceptance predicate gates the commit: $\(s_{t+1} = \begin{cases} \text{Commit}(w_t \oplus \Delta_t,\, y_t,\, \Gamma_p) & A_p(y_t) = 1 \\ \text{RejectLog}(s_t, \Delta_t, y_t) & A_p(y_t) = 0 \end{cases}\)$

Git as substrate. Every accepted attempt is a git commit with evaluator evidence attached via git notes; failed attempts are logged. git diff --cached exposes proposed changes; git log recovers trajectories. The repository history doubles as the experience buffer with no separate datastore.

Token efficiency. A persistent model session is reused across iterations so the harness, project pack, and stable sources are served from the provider's prompt cache; only the current diff, latest evaluator output, and agent response consume fresh tokens.

Key Contributions

• HORIZON framework: hosts each hardware design problem as an isolated, version-controlled, automatically-evaluated git worktree—agnostic to the underlying LLM backbone.
• 100% benchmark sweep: a single hands-free agentic loop closes all evaluated RTL benchmarks to 100% pass rate.
• Trace analysis: characterizes convergence difficulty, iteration budget, and token consumption across suites; shows 91% of ~210M total tokens are cached input.
• Generalization of repository-scale self-evolution from EDA software (SATLUTION, ABCEvo) to hardware design artifacts (RTL sources, testbenches, verification artifacts).

Results

• Overall: 100% pass rate across all 12 evaluated suites/categories (Table 1); one ChipBench task non-passing due to a benchmark specification–harness defect, not agent failure.
• Iteration 0 aggregate: 47.8% pass rate (Verilog-Eval-v2: 86.2%; CVDP subset: 23.9%; CID 002: 3.2%; CID 013: 3.8%).
• Convergence speed: RTLLM-2.0 and Verilog-Eval reach 100% in 2 iterations; ChipBench in 5; CVDP categories range from 1 (CID 014) to 82 (CID 002) iterations.
• Token budget: ~210M total tokens; CID 002 alone accounts for 56.0M (26.7%); legacy suites (ChipBench + RTLLM + Verilog-Eval) together use only 6.0M (2.9%).
• CID 013 (checker generation, commercial EDA): lowest first-iteration rate (3.8%), but converges near-linearly to 100% by iteration 19—showing low first-pass accuracy does not imply slow convergence.

Limitations

• Benchmarks are controlled proxies: the paper explicitly disclaims that agentic hardware design is solved; broader scope (architecture exploration, physical design, microarchitecture) is a framework goal without empirical validation.
• Fixed backbone, no RL: agent policy (GPT-5.3) is not updated; the trace formulation is designed to support offline agent-RL but no training is performed.
• Synthesis QoR not reported: the \(\Delta\text{QoR}\) reward component is left to future work.
• Commercial EDA dependency: CID 012–014 require a commercial simulator, limiting reproducibility.
• High token cost on hard tasks: CID 002 requires 82 iterations and 56M tokens; economic cost analysis (pricing, infrastructure, parallelism) is excluded.
• Single-agent mode only: all results are in single-agent mode; multi-agent or population-based variants are not evaluated.

Relevance to Agentic AI / LLM Agents

HORIZON is a concrete instance of the emerging repository-native agent paradigm—alongside EvoGit and Git Context Controller—where git operations (branch, commit, diff, log) serve as the agent's memory, state management, and experience replay substrate rather than an external datastore. The paper demonstrates that executable feedback loops with correctness gates can drive an agent from near-zero first-attempt accuracy (3.2%) to complete benchmark coverage, directly quantifying the gap between single-turn generation and agentic iteration. The semi-MDP trace formulation (\(\tau_{0:D} = \{(s_t, a_t, r_t, s_{t+1}, y_t)\}\)) provides a replayable experience buffer explicitly designed for future offline agent-RL or curriculum learning, making HORIZON a methodology contribution as much as a benchmark result. For researchers tracking LLM agents, this work extends the AlphaEvolve/SATLUTION lineage to a new domain and provides a reproducible harness pattern (Markdown → project pack → hands-free loop) that is domain-agnostic.