Skip to content

David vs. Goliath: Can Small Models Win Big with Agentic AI in Hardware Design?

🕒 Published (v1): 2025-12-04 18:37 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper asks whether small language models (SLMs, ≤20B parameters) paired with a purpose-built agentic scaffold can match frontier LLM performance on hardware RTL design tasks. Evaluating on NVIDIA's CVDP benchmark, the authors show 30–140% relative gains from agentic scaffolding, with SLMs surpassing GPT-o4-mini on select task categories (cid007: 51.25% vs. 44.74%) at 8–10× lower energy cost.

Problem

Prior work either benchmarks SLMs in single-shot mode (showing 35–45% pass rates vs. GPT-4's 58%) or deploys multi-agent systems exclusively on large commercial models. No work had tested whether SLM performance ceilings reflect inherent capability or simply inadequate scaffolding—leaving open whether sophisticated agentic frameworks can close the gap without the energy/cost overhead of frontier models.

Method

The authors build a five-agent closed-loop pipeline tailored to SLM constraints:

  1. Planning and Pre-processing Agent (PPA): Keyword/structure-based context retrieval from CVDP, problem decomposition into sub-goals, and packaging into a context bundle.
  2. SLM-Aware Prompt Engineering Agent (SPEA): Injects structural attention anchors (ROLE, TASK, REQUIREMENTS, CONSTRAINTS, GENERATE), selects few-shot examples by task similarity, and enforces a token budget (40% task / 40% context / 20% examples) within 8K–32K context windows.
  3. CodeGen Agent (CA): Runs SLM inference with reasoning suppression to maximize output tokens for code; extracts Verilog via pattern-matched parsing.
  4. Validation Agent (VA): Three-stage pipeline—RTL linter (syntax), regex-based I/O port checker (detects "zombie ports"), and CocoTB functional test runner.
  5. Adaptive Feedback Agent (AFA): Categorizes errors into seven classes (syntax, undeclared, type, width, latch, timing, novel); scores outputs with a composite metric (60% functional correctness + 20% compilation + 10% port completeness + 10% structure); triggers rollback to highest-scoring prior version if quality regresses.

The pipeline iterates for up to five rounds. Evaluated models: SmolLM2 (1.7B), NemotronMini (4B), Granite-4 (3B), DeepSeek-R1 (7B), Phi-3.5-mini-instruct, GPT-OSS (20B), with GPT-o4-mini as the LLM baseline.

Key Contributions

  • First systematic evaluation of SLMs (<20B) inside a sophisticated agentic framework on the CVDP benchmark.
  • Novel SLM-aware agentic framework: structured keyword injection, token budget management, zombie-port detection, error categorization, and score-based rollback.
  • Empirical validation of "strategy over scale": well-designed scaffolding enables SLMs to match or exceed frontier LLMs on certain task categories.
  • Planned open-source release of the framework.

Results

  • Code generation (cid007—RTL code improvement/linting): DeepSeek-R1 and Granite-4 achieve 51.25% and 48.75% pass@1, respectively, surpassing GPT-o4-mini at 44.74%; roughly 100% relative improvement over single-shot for these models.
  • Agentic vs. single-shot across all code generation tasks: 30–140% relative improvement for SLMs when adding the agentic scaffold.
  • Code comprehension (cid009/cid010—Q&A tasks): phi-3.5-mini-instruct and DeepSeek-R1 reach 82–92% accuracy, matching LLM performance.
  • Code comprehension (cid008—testbench-to-test-plan): phi-3.5-mini-instruct and DeepSeek-R1 substantially exceed GPT-4o baseline on structured reconstruction tasks.
  • Energy: DeepSeek-7B consumes 0.55 ± 0.06 Wh vs. 7.24 ± 1.67 Wh for GPT-4o-mini per query (≈13× reduction).
  • SLMs still underperform on cid004 (RTL code modification); isolated anomalies on cid002/cid003/cid016 for Nemotron and Granite-4.

Limitations

  • Evaluation limited to CVDP's Non-Agentic Code Generation and Comprehension categories; CVDP's native Agentic Code Generation tasks are excluded.
  • Iteration cap of five rounds; no ablation across different caps.
  • No fine-tuning or domain adaptation of the SLMs—purely prompting-based scaffolding.
  • Small model pool (four SLMs); task-specialized SLMs left for future work.
  • Results are benchmark-specific (CVDP); generalizability to real SoC design flows is unconfirmed.
  • BLEU-based evaluation for cid006/cid008 is a weak proxy for functional correctness in schema-constrained generation.

Relevance to Harnesses / Meta-Harnesses

This paper is a concrete instantiation of a domain-specific meta-harness: a five-agent closed-loop orchestrator that wraps a base model with planning, prompt engineering, generation, multi-stage validation, and feedback-driven rollback—exactly the pattern central to harness/meta-harness research. The score-based rollback and error-categorization loop are directly analogous to self-repair meta-harnesses in software engineering. The work demonstrates quantitatively that harness design (task decomposition, structured prompting, validation coupling) can be load-bearing—responsible for 30–140% performance gains—making it a strong empirical case for harness contribution being separable from and competitive with model scale. The SPEA's token budget management and the AFA's composite scoring function are reusable harness primitives worth tracking.