Skip to content

Agentic Hardware Design as Repository-Level Code Evolution

🕒 Published (v1): 2026-06-26 17:21 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HORIZON is a self-evolving agent framework that compiles a structured Markdown harness into a "project pack" containing an executable evaluator, acceptance predicate, and git/runtime policy, then runs a hands-free loop that evolves an isolated git worktree until all benchmark tasks pass. Applied to RTL hardware design, it achieves 100% pass rate across ChipBench, RTLLM-2.0, Verilog-Eval, and nine CVDP categories from a starting aggregate of 47.8%. The key claim is that git is not bookkeeping but the agent's state substrate, trace buffer, and experience store simultaneously.

Problem

Single-turn code generation fails on RTL because correctness requires cycle-accurate, bit-precise simulation feedback; plausible Verilog is insufficient. Prior iterative/agentic RTL systems (AutoChip, RTLFixer, VerilogCoder, MAGE) build bespoke generate-and-repair pipelines for individual modules and do not generalize across task types or provide a replayable, versioned trace. Repository-level self-evolution (AlphaEvolve, SATLUTION, ABCEvo) had only been applied to EDA software, not to the hardware artifacts engineers produce.

Method

The user writes a structured Markdown harness \(m\) specifying objective, domain knowledge, evaluator specification, and acceptance predicate. A bootstrap agent \(G_\phi\) compiles this into a project pack:

\[p = G_\phi(m) = (\pi_{\text{agent}},\; E_p,\; A_p,\; \Gamma_p,\; \Omega_p)\]

where \(\pi_{\text{agent}}\) is the agent policy prompt, \(E_p\) is the executable evaluator (compile + simulate + coverage/assertion checks), \(A_p\) is the pass/fail gate, \(\Gamma_p\) is the git/artifact policy, and \(\Omega_p\) holds domain skills. The agent loop then operates over an isolated git worktree \(W\): at each outer step \(t\) it proposes a patch \(\Delta_t\), runs the evaluator \(y_t = E_p(w_t \oplus \Delta_t)\), and either commits (if \(A_p(y_t)=1\)) or logs rejection. Repository state is formalized as

\[s_t = (\text{tree}(w_t),\; p,\; z_t,\; \ell_{\leq t},\; \mu_t)\]

Git diffs expose proposed changes, commits are checkpoints with attached evaluator evidence via git notes, and git log reconstructs the full replay trace. A persistent LLM session (GPT-5.3) maximizes prompt-cache reuse: the harness, project pack, and stable sources are served from cache; marginal billed tokens are dominated by the current diff and latest evaluator output. The reward vector is \(r_t = (\Delta\text{pass},\;\Delta\text{coverage},\;\Delta\text{QoR},\;-\text{tokens},\;-\text{time})\); only \(\Delta\text{pass}\), \(\Delta\text{coverage}\), and \(-\text{tokens}\) are reported in this work.

Key Contributions

  • HORIZON framework: hosts arbitrary hardware-design tasks (RTL, EDA software, methodology) as isolated, version-controlled, auto-evaluated git repositories driven by a Markdown harness.
  • Git-native trace substrate: diffs, commits, notes, and logs serve as the agent's experience buffer and replay record without any separate datastore.
  • 100% benchmark sweep: a single hands-free agentic loop achieves 100% pass rate on ChipBench, RTLLM-2.0, Verilog-Eval, and all nine evaluated CVDP code/verification-generation categories.
  • Convergence and cost analysis: reports per-benchmark iteration counts and token consumption, showing ~91% cache hit rate and that a low first-iteration pass rate does not imply slow convergence (CID 013: 3.8% → 100% in 19 near-linear iterations).

Results

  • Overall: 47.8% aggregate pass rate at iteration 0 → 100% final, across all evaluated suites (Table 1).
  • Legacy suites: RTLLM-2.0 and Verilog-Eval reach 100% in 2 iterations; ChipBench in 5 iterations (one miss traced to a benchmark specification–harness defect, not agent failure).
  • CVDP hardest categories: CID 002 (RTL code completion) starts at 3.2%, requires 82 iterations and 56.0M tokens; CID 013 (checker generation) starts at 3.8%, reaches 100% in 19 iterations with near-linear trajectory.
  • Token consumption: 210M total; ~91% cached input; three legacy suites use 6.0M (2.9%), nine CVDP categories use 203.9M (97.1%); CID 002 alone accounts for 26.7%.
  • No baseline comparison in the single-agent campaign sense; prior works are positional (first-attempt Pass@1) rather than convergence-based.

Limitations

  • Benchmarks (ChipBench, RTLLM, Verilog-Eval, CVDP) are controlled proxies; full chip design encompasses architecture exploration, microarchitecture, physical design, and EDA methodology, none of which are evaluated.
  • 100% completion does not imply quality-of-results (QoR) or synthesis metrics; \(\Delta\text{QoR}\) is explicitly deferred to future work.
  • Convergence efficiency is the identified bottleneck (not correctness), but no method for reducing the long tail (e.g., CID 002 at 82 iterations) is proposed.
  • CID 012–014 require commercial EDA simulation, limiting reproducibility.
  • Single-agent mode only; multi-agent parallelism is not explored.
  • Agent backbone is fixed (GPT-5.3); no ablation over backbone choice or harness quality.
  • One ChipBench task fails due to a specification–harness mismatch in the original benchmark, revealing that harness correctness is load-bearing and not verified automatically.

Relevance to Harnesses / Meta-Harnesses

HORIZON is directly a harness-compilation meta-harness: the user's Markdown specification is itself the input artifact that a bootstrap agent transforms into an executable task instance (project pack), making the harness a first-class programmatic object rather than a prompt template. The formalization \(p = G_\phi(m) = (\pi_{\text{agent}}, E_p, A_p, \Gamma_p, \Omega_p)\) is a concrete schema for what a harness must contain to support autonomous agent loops, directly relevant to anyone designing meta-harnesses that generate or instantiate task harnesses programmatically. The git worktree as trace buffer pattern—where the harness's acceptance predicate doubles as the convergence criterion and the repository history becomes the experience log—demonstrates a lightweight, infrastructure-free design for harness-driven self-improvement loops. This complements AlphaEvolve/SATLUTION-style meta-harnesses by showing the pattern generalizes from algorithm/software evolution to artifact design domains.