RTL-BenchMT: Dynamic Maintenance of RTL Generation Benchmark Through Agent-Assisted Analysis and Revision¶

🕒 Published (v1): 2026-05-15 02:17 UTC · Source: Arxiv · link

Why this paper was selected

RTL-BenchMT: agentic dynamic maintenance prevents benchmark staleness in EDA research

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

RTL-BenchMT is a multi-agent framework that automates the maintenance of RTL generation benchmarks by identifying and revising flawed benchmark cases and detecting LLM overfitting. It orchestrates specialized agents through three processes—failure analysis, benchmark revision, and overfitting detection—with human engineers reviewing and approving final changes. Applied to VerilogEval, RTLLM, and CVDP, it finds roughly 10% of benchmark cases are flawed.

Problem¶

LLM-based RTL generation benchmarks suffer from two compounding issues: (1) flawed cases where mismatches between design descriptions, testbenches, and reference code cause correct LLM outputs to be scored as failures, leading to misleading capability estimates; and (2) overfitting, where public benchmark exposure allows LLMs to memorize or exploit superficial description patterns, inflating performance numbers. Manual identification and correction of both problems across large benchmarks is prohibitively labor-intensive.

Method¶

RTL-BenchMT organizes agents into three sequential processes:

Failure Analysis Process: A failure analysis agent runs a structured three-iteration reasoning template (Algorithm 1). Iteration 1 compares generated vs. reference code (COMPARE_CODES). Iteration 2 checks whether the generated code correctly implements the design description (CHECK_INSTRUCTION). Iteration 3, triggered only when the generated code is found correct, cross-examines the description against the testbench (REASON) to isolate description-level flaws. The initial observation \(O_0\) is evaluation results from running multiple LLMs; each subsequent observation is \(O_i = \text{Env}(A_i)\) and each thought is \(T_i = \text{Agent}(O_{i-1})\).
Benchmark Revision Process: A description revision agent proposes a corrected description; a description review agent validates it against two strict rules—no destruction of basic semantics, no leakage of testbench implementation details.
Overfitting Detection Process: A description update agent rewrites descriptions across pre-defined style templates (Technical/Formal, Educational/Tutorial, Problem-Solving, Specification/Documentation) while preserving semantics and functionality, formulated as \(T_\text{Plan} = \text{Agent}(D_\text{design}, \{S_i\});\ D_\text{variation} = \text{Agent}(T_\text{Plan})\). A model is flagged as overfitted if it passes on the original description but fails on a rewritten variant.

Agents run in a lightweight host environment; EDA tools (VCS, Iverilog) execute inside a Dockerized license-server environment accessed through an agent–tool interface that handles connection, file upload, compilation, simulation, and result download.

Key Contributions¶

A three-process agentic framework (failure analysis → benchmark revision → overfitting detection) for continuous benchmark maintenance with human-in-the-loop approval.
A structured three-iteration analysis reasoning template that isolates benchmark flaws from genuine LLM capability gaps.
A taxonomy of benchmark flaws across three categories: syntax ambiguity (undefined module names, unclear port types, code-sample syntax errors), functional behavior ambiguity (register initialization, trigger conditions, missing sub-module implementations), and diagram ambiguity (KMap input mismatch, incomplete FSM transition conditions).
Quantitative overfitting detection per case and per model via semantics-preserving description variation.
Revised, open-sourced benchmark cases for VerilogEval, RTLLM, and CVDP.

Results¶

Approximately 10% of test cases across all evaluated benchmarks (VerilogEval, RTLLM, CVDP) are identified as ambiguous/flawed; the rate of constantly-failing cases strongly correlates with the number of identified flaws, suggesting benchmark error rather than model deficiency is a major source of failures.
Re-evaluation on the corrected benchmark shows GPT-4o exhibits obvious performance improvement, while Claude-3.7 shows minor variance, suggesting GPT-4o was more sensitive to description flaws.
Flaw distribution differs by benchmark: VerilogEval flaws are primarily functional-behavior and diagram types (no syntax flaws); RTLLM flaws exclude diagram ambiguity; CVDP (cid002/cid003) introduces syntax-category flaws not seen in other benchmarks.
(Precise per-benchmark counts from Table 3 are in the truncated portion of the paper and are not reported here to avoid fabrication.)

Limitations¶

Human engineers must still review and approve all agent-proposed revisions; the framework reduces but does not eliminate manual effort.
Overfitting detection relies on a pass/fail binary per rewritten description—it does not quantify degree of overfitting or distinguish memorization from pattern exploitation.
The description-variation strategy assumes semantics can be fully preserved across style rewrites; subtle semantic drift in agent-generated variations could introduce new benchmark errors.
Evaluation is limited to three benchmarks (VerilogEval, RTLLM, CVDP); generalizability to other RTL or hardware-description benchmarks is unverified.
The EDA environment requires a Dockerized license server, limiting portability.

Relevance to Harnesses / Meta-Harnesses¶

RTL-BenchMT is a canonical example of a meta-harness: rather than running evaluations, it wraps and maintains the evaluation infrastructure itself, orchestrating agents that audit, revise, and diversify benchmark content. Its architecture—a manager agent routing to specialized sub-agents (analysis, revision, review, update), each following a thought-action-observation loop over a shared tool interface—directly instantiates the meta-harness pattern of agents operating on the outputs of other agents. The explicit human-in-the-loop checkpoint before benchmark changes are committed models a gated approval pattern relevant to any harness system managing shared evaluation state. For researchers building harnesses over evolving benchmarks, RTL-BenchMT demonstrates how to decompose benchmark quality control into automatable sub-tasks with well-defined interfaces between agent roles.