Skip to content

ChipCraftBrain: Validation-First RTL Generation via Multi-Agent Orchestration

🕒 Published (v1): 2026-04-21 17:20 UTC · Source: Arxiv · link

Why this paper was selected

Validation-first RTL generation via multi-agent orchestration; correctness improvement

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ChipCraftBrain is a validation-first RTL generation framework that coordinates six specialized LLM agents via a PPO-trained orchestration policy over a 168-dimensional state vector, combined with a hybrid symbolic-neural router and a three-stage validation pipeline (lint → simulation → synthesis). It achieves 98.7% pass@1 on VerilogEval-Human and 94.7% on the 302-problem non-agentic CVDP subset, outperforming all published baselines on the latter by 36–60 percentage points per category.

Problem

Single-shot LLM RTL generation saturates at 60–65% functional correctness on VerilogEval. The strongest prior multi-agent system, MAGE, reaches 95.9% but has three specific gaps: (a) no synthesis-quality signal in candidate scoring, (b) raw waveform dumps for debug feedback, and (c) high API cost ($0.06/problem from 20 parallel Claude Sonnet calls). Workflow-search approaches like VFlow (83.6%) discover a static workflow that cannot adapt per-problem complexity or error pattern. No prior agentic system had published results on NVIDIA's CVDP industrial benchmark.

Method

The system implements a three-tier dispatch before any LLM call:

  1. Algorithmic tier: K-map and truth-table problems are solved via Quine-McCluskey minimization—zero API cost, perfect accuracy.
  2. Rule-based tier: Waveform/timing problems route to a dedicated Waveform agent (Opus 4.6) via keyword detection; it enforces a six-step structured trace-analysis protocol.
  3. RL-orchestrated tier: For all other RTL tasks, a PPO actor-critic policy (shared \(2\times256\) MLP trunk with LayerNorm/ReLU) selects from four specialized agents and configures a hybrid discrete-continuous action: $\(a = (a_{\text{agent}}, a_{\text{focus}}, a_T, a_{\text{tokens}}, a_{\text{RAG}}, a_{\text{retry}})\)$ The 168-dimensional state encodes task metadata, iteration progress, structured error signals, code metrics, agent-selection history, and a deterministic per-spec identifier for reproducibility. The multi-component reward is: $\(R = R_{\text{term}} + R_{\text{eff}} + R_{\text{qual}} + R_{\text{prog}}\)$ where \(R_{\text{term}} \in \{+100, +60, -50\}\) (sim pass / lint pass / failure), \(R_{\text{eff}}\) penalizes token cost, \(R_{\text{qual}}\) rewards low LUT count and timing closure, and \(R_{\text{prog}}\) rewards per-stage advances and per-error eliminations. Training runs online PPO from episode 20 onward after a 20-episode heuristic warm-start; the released checkpoint covers 57 episodes with \(\varepsilon\) decayed from 0.3 to 0.225.

The four RL-orchestrated agents differ in model (Opus 4.6 vs. Sonnet 4.6), temperature (0.3–0.7), and RAG depth (3–15 chunks): Genius (Opus, \(T=0.7\), 15 chunks, full context), Fast (Sonnet, \(T=0.5\), 3 chunks), Debug (Opus, \(T=0.3\), 5 chunks, error-focused retrieval), Optimize (Sonnet, \(T=0.4\), 4 chunks, synthesis-focused). Knowledge retrieval draws from a 321-entry curated knowledge base (RTL patterns, architecture templates, optimization strategies) plus 971 indexed open-source reference implementations across 14 hardware domains. A spec-guidance registry of 59 detectors injects problem-type-specific technical briefs before generation. Complex multi-component designs are detected and decomposed into 4–8 dependency-ordered sub-modules with cross-module port synchronization before the iterative loop. An alternative MPC planner (64 candidate sequences, 3-step horizon, trained world model) is implemented and evaluated alongside PPO.

Key Contributions

  • PPO-trained orchestrator over a 168-dim state selecting among 6 specialized agents plus continuous temperature, RAG depth, and token-budget parameters
  • Hybrid symbolic-neural router: deterministic Quine-McCluskey K-map solver and structured Waveform agent eliminate two problem classes from LLM generation entirely
  • Focus-aware RAG from dual knowledge sources (curated 321-entry KB + 971 reference implementations) with RL-controlled retrieval depth \(k \in [3, 20]\)
  • Hierarchical decomposition engine generating dependency-ordered sub-modules with interface synchronization, enabling SoC-scale generation where monolithic approaches fail
  • Three-benchmark evaluation (VerilogEval / CVDP / ChipBench) spanning simple modules to industrial IP

Results

  • VerilogEval-Human (156 problems): 98.7% pass@1 best run (154/156); range 96.15–98.72% across 7 runs; vs. MAGE 95.9%, ChipAgents 97.4% (self-reported)
  • CVDP non-agentic non-commercial code-generation subset (302 problems, 5 categories): 94.7% pass@1 (286/302) with 5-iteration refinement; vs. Claude 3.7 Sonnet single-shot baseline 33.56%—per-category lifts of 36–60 pp; leads 3 of 4 shared categories vs. ACE-RTL (NVIDIA's own agentic system using up to 150 total attempts vs. ChipCraftBrain's ~5)
  • K-map solver: 3/3 problems solved with zero API cost
  • Mercury 2 diffusion backbone routed through the same pipeline: 82.7% pass@1 on VerilogEval-Human at ~5 s/problem
  • MPC planner vs. PPO (single controlled run, VerilogEval-Human): 97.4% vs. 96.8%, 1.8% fewer average iterations
  • RISC-V SoC case study: 8/8 lint-passing modules (689 LOC), FPGA validated; monolithic generation fails entirely

Limitations

  • PPO training set overlaps with VerilogEval-Human evaluation benchmark; memorization cannot be ruled out—authors explicitly treat policy as a proof-of-concept and defer clean train/test separation to future work
  • 98.7% headline is best-of-7 runs; single-run average is 96.15–98.72% with uncharacterized cross-run variance
  • CVDP testbench/assertion categories (cid012–cid014) require Cadence Xcelium and are excluded from evaluation
  • ChipBench results are cited for baselines (MAGE 37.4%, Claude Opus 30.7%) but ChipCraftBrain's own ChipBench numbers are not reported in the provided text
  • MPC planner comparison is a single controlled run; cross-run variance not reported for MPC

Relevance to Harnesses / Meta-Harnesses

ChipCraftBrain is a domain-specific meta-harness whose central concern is learned orchestration: a trained PPO policy replaces hand-written dispatch rules to select which sub-agent to invoke and how to configure it (temperature, retrieval depth, token budget) given a structured execution-state vector. The three-tier dispatch (algorithmic → rule-based → RL) is a concrete instantiation of tiered meta-harness architecture—offloading deterministic sub-problems to zero-cost solvers before engaging costly LLM agents. The validation pipeline (lint → simulation → synthesis) acts as the harness's structured feedback channel, converting opaque tool outputs into reward signals and typed error features that drive iterative refinement. The MPC planner alternative shows that the orchestration layer is modular and substitutable, a design property directly relevant to building general-purpose meta-harnesses where the routing strategy can be swapped without changing agent implementations.