Skip to content

PEFA-AI: Advancing Open-source LLMs for RTL generation using Progressive Error Feedback Agentic-AI

🕒 Published (v1): 2025-11-06 00:19 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PEFA-AI is a multi-agent agentic framework for automated RTL (Verilog) generation that introduces a Progressive Error Feedback Agent (PEFA) system: instead of dumping full error logs back to the LLM, it stages and summarizes feedback across up to 4 iterative correction loops, using a small Llama-8B log-summarizer as a filtering agent. The system achieves state-of-the-art pass rates on VerilogEval and RTLLM1.1 while using only 4 LLM calls versus the pass@20 baseline's 20 calls.

Problem

Conventional self-correcting LLM pipelines for RTL generation feed raw, often verbose error logs back to the main model, causing context overload, hallucinations, and divergence. Human-in-the-loop systems require manual prompt refinement each iteration, limiting automation. Prior agentic RTL frameworks (MAGE, VerilogCoder) lack rigorous token-efficiency benchmarking and use heavily tuned temperature heuristics, making fair comparison difficult.

Method

The framework is built on AutoGen's Group_chat_manager and consists of five coordinated agents:

  1. Master agent — parses the user's natural-language prompt and test bench; augments the test bench with $monitor and VCD dump commands without exposing it to the LLM (black-box IP protection).
  2. Code generator agent — calls the backend LLM zero-shot to produce Verilog.
  3. Code executor agent — runs Verilator linting (-Wall), iverilog compilation, and simulation; aborts at first failure and captures the stack trace.
  4. Log summarizer agent (PEFA core) — implements progressive feedback via a Llama-8B small model:
  5. Loop 1: converts VCD to CSV and surfaces signal mismatches around the first divergence point.
  6. Loops 2–4: LLM-summarizes the error log into a few lines before feeding it back.
  7. Full error logs are never passed directly to the main LLM.
  8. Summary generator — synthesizes the full group chat into a user-facing explainability report.

Context is managed aggressively: the group chat history is cleared at each loop; only the most recent passing-compilation code plus the latest summarized log are kept in the main LLM's context window (queue-based popping).

Key Contributions

  • Novel progressive error feedback staging (syntactic failure → first-mismatch VCD excerpt → summarized functional error), preventing token explosion and hallucination cascades.
  • Hybrid agents combining programmatic tools (Verilator, iverilog, VCD parser) with LLMs, treating the test bench as a black-box to preserve IP.
  • Context manipulation protocol (message queue, history clearing) that prevents group-chat context poisoning in multi-turn agentic loops.
  • Empirical demonstration that 4-call agentic correction outperforms pass@20 (20 independent calls) on both open- and closed-source models.
  • Narrows open-source vs. closed-source pass-rate gap (e.g., Llama-3.1:70B improves +134.78% on Spec-to-RTL with PEFA vs. non-agentic baseline).

Results

  • VerilogEval Code Completion (pass@20 equivalent, N=20 agentic runs):
  • Claude-3.5-Sonnet Progressive: 85.9% vs. non-agentic 78.21%
  • GPT-4o Progressive: 83.97% vs. 75.0%
  • Llama-3.1:70B Progressive: 79.49% vs. 68.58%
  • DeepSeekCoder:33B Progressive: 64.1% vs. 46.9%
  • VerilogEval Spec-to-RTL (removing 7% erroneous cases):
  • Claude-3.5-Sonnet: 97.95% (vs. VerilogCoder GPT-4 Turbo: 94.8%)
  • GPT-4o: 86.98%
  • Llama-3.1:70B: 84.00%
  • RTLLM1.1:
  • GPT-4o Progressive: 86.83% vs. non-agentic 71.81%
  • Claude-3.5-Sonnet Progressive: 87.10% vs. 77.27%
  • Token efficiency: 4 agentic calls use ~3,560 tokens (Claude-3.5-Sonnet) vs. 20 × 440 = 8,800 tokens for pass@20 — a ~2.5× reduction.
  • Progressive feedback beats simple feedback by +4.5% (Code Completion) and +7.8% (Spec-to-RTL) for Llama-3.1:70B; +5.8% / +3.0% for Claude-3.5-Sonnet.

Limitations

  • Maximum of 4 feedback loops is hard-coded; no dynamic termination based on convergence confidence.
  • RTLLM1.1 lacks pre-built test benches; ground-truth test bench creation required manual expert effort, introducing potential annotation bias.
  • Fine-tuning the log-summarizer agent to suggest targeted corrections (rather than merely summarizing errors) remains an open challenge.
  • Token counts exclude calls to the Llama-8B log summarizer, making the full compute cost comparison with pass@20 incomplete.
  • PPA-aware RTL generation is deferred to future work; current evaluation covers only functional correctness.
  • Robustness to temperature variation is demonstrated only in an appendix; ablations across prompt styles or hardware design complexity are absent.

Relevance to Harnesses / Meta-Harnesses

PEFA-AI is a concrete instance of a task-specific meta-harness: it wraps multiple heterogeneous agents (LLMs, static analyzers, simulators, small summarizer models) under a shared orchestration layer (AutoGen Group_chat_manager) with explicit state-machine control flow. The progressive feedback staging — where the harness decides what information to surface to the LLM at each loop iteration rather than passing raw tool outputs — is a direct example of the context management challenge central to meta-harness design. The context-clearing and queue-based message filtering strategies address the general problem of preventing long-running agentic loops from accumulating irrelevant history, a concern applicable to any multi-turn harness. The paper also provides empirical ablation data on how harness-level design choices (feedback granularity, history management) dominate raw LLM capability — a key insight for anyone designing orchestration layers.