Skip to content

Automated Multi-Agent Workflows for RTL Design

🕒 Published (v1): 2025-09-24 14:44 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VeriMaAS is a multi-agent framework that automatically composes agentic operator workflows for RTL/Verilog code generation by integrating formal verification feedback (Yosys synthesis logs, OpenSTA timing/power reports) directly into a cascading workflow controller. It tunes cascade thresholds on ~500 examples rather than thousands, achieving 5–12% pass@k gains over fine-tuned baselines at moderate token overhead.

Problem

RTL code generation with LLMs faces a trilemma: fine-tuning requires large GPU budgets and domain-specific data, frontier reasoning models shift cost to inference, and general-purpose multi-agent workflow generators (MaAS, AFlow, Flow) were designed for QA/math/coding domains and do not exploit the rich formal feedback signals available in HDL synthesis toolchains. A domain-specific automated workflow generator that leverages EDA tool output was missing.

Method

VeriMaAS defines a solution space \(\mathcal{O} = \{O_{\text{I/O}}, O_{\text{CoT}}, O_{\text{ReAct}}, O_{\text{SelfRefine}}, O_{\text{Debate}}\}\) ordered by complexity. A cascading controller \(C\) selects operators sequentially: at each stage \(c\), it runs \(K=20\) Verilog candidates through Yosys (synthesis, area) and OpenSTA (timing, power), computes a confidence score \(s_c\) as the fraction of failing designs, and escalates to the next operator if \(s_c\) exceeds a stage threshold \(\tau_c\). The threshold vector \(T = \{\tau_1, \ldots, \tau_C\}\) is optimized via:

\[\max_T \mathbb{E}_{(q,a)\sim\mathcal{D}}\left[U(T;q,a,\mathcal{O}) - \lambda \cdot C(T;q,a,\mathcal{O})\right]\]

where \(U\) is pass@k and \(C\) is average tokens per query (\(\lambda = 10^{-3}\)). Thresholds are set as the 20th/40th/60th/80th percentiles of failure-count distributions over 500 sampled training points — no gradient updates. For PPA-aware optimization, the cost term is replaced with Yosys-reported area, and an oracle LLM pre-selects the 20 designs where RTL changes most affect PPA.

Key Contributions

  • Cascading agentic controller that uses EDA synthesis/verification failure rates as a direct signal for operator escalation, replacing gradient-based or RL-based policy learning.
  • Order-of-magnitude reduction in supervision: threshold tuning on ~500 examples vs. tens of thousands for full fine-tuning.
  • PPA-aware re-optimization by swapping the cost term in the objective — no retraining required.
  • Empirical validation across proprietary (GPT-4o-mini, o4-mini) and open-weight (Qwen2.5-7B/14B, Qwen3-8B/14B) models on VerilogEval and VeriThoughts benchmarks.

Results

  • vs. instruct baselines: Qwen2.5-7B pass@1 on VeriThoughts: 44.90 → 56.62 (+11.72%); Qwen2.5-14B pass@1 on VeriThoughts: 67.89 → 74.24 (+6.35%).
  • vs. fine-tuned RTLCoder-7B: VeriMaAS on Qwen2.5-7B achieves 29.10 pass@1 on VerilogEval vs. RTLCoder's 34.60 (still below), but 56.45 pass@10 vs. RTLCoder's 45.50.
  • vs. fine-tuned VeriThoughts-14B: VeriMaAS on Qwen2.5-14B achieves 74.24 pass@1 vs. 78.50 (fine-tuned wins on pass@1), but 95.78 vs. 92.10 pass@10.
  • Token cost: VeriMaAS sits near CoT overhead (1.20–2.52× base), well below SelfRefine (2.22–2.90×).
  • PPA optimization: Area reductions up to 28.79%, delay reductions up to 24.58%; power trade-offs vary by model.
  • Closed-source models (GPT-4o-mini, o4-mini) show smaller but consistent gains (+0.24–2.45% pass@1).

Limitations

  • Threshold tuning still requires a labeled training set with ground-truth Verilog solutions; it just needs fewer (~500) examples.
  • PPA-aware mode relies on an expensive oracle LLM (o4) to pre-select candidate designs, limiting scalability.
  • Pass@10 on VerilogEval shows marginal drops for some models across all operator strategies, suggesting benchmark-specific variability not fully explained.
  • Controller is a fixed linear cascade; no tree-search or RL-based exploration (noted as future work).
  • Evaluation limited to open-source PDK (Skywater 130nm); commercial EDA tools and PDKs not yet integrated.

Relevance to Harnesses / Meta-Harnesses

VeriMaAS is a domain-specific meta-harness that automatically composes operator pipelines from a predefined operator library, directly instantiating the meta-harness pattern of selecting and chaining sub-agents based on runtime feedback. Its key architectural contribution to the harness design space is using formal tool feedback (synthesis logs) as the harness's routing signal rather than LLM self-assessment or reward models — a concrete example of environment-grounded orchestration. The threshold-based controller parameterization (\(T\)) mirrors the general meta-harness challenge of learning when to escalate compute, solved here with percentile calibration instead of gradient descent. This work extends the MaAS/AFlow/Flow lineage into a specialized synthesis domain, demonstrating that domain-specific feedback channels can replace expensive training signals in automated workflow generation.