Skip to content

AutoFSM: A Multi-agent Framework for FSM Code Generation with IR and SystemC-Based Testing

🕒 Published (v1): 2025-12-12 09:15 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AutoFSM is a multi-agent framework for LLM-based finite state machine (FSM) Verilog code generation that introduces a JSON intermediate representation (IR) to reduce syntax errors and SystemC-based automatic testbench generation to improve debugging. It outperforms the open-source MAGE framework by up to 11.94% in pass@1 and reduces syntax error rate by up to 17.62% on a new FSM-specific benchmark (SKT-FSM).

Problem

LLMs generating Verilog for FSM control logic produce ~55% syntax errors, lack automated testbench generation (relying on manually written benchmark test programs), and existing multi-agent frameworks (MAGE, VerilogCoder) provide limited diagnostic feedback when errors occur—mixing syntax and functional errors makes fault localization difficult. Existing benchmarks (VerilogEval) contain too few and too simple FSM examples to meaningfully evaluate FSM-oriented generation.

Method

AutoFSM chains six specialized agents—FSMExtractor, Verifier, Coder, Tester, Fixer, Judger—built atop MetaGPT:

  1. IR-based generation: FSMExtractor converts natural language specs into a JSON IR with explicit field definitions (avoiding YAML's reserved-keyword ambiguities). A custom json2yaml converter feeds this into fsm2sv to deterministically compile valid Verilog, bypassing direct Verilog generation by the LLM entirely.
  2. Verifier loop: A Verifier agent checks JSON-to-intent alignment before Verilog generation; inconsistencies cycle back to FSMExtractor.
  3. SystemC-based differential testing: Tester agent generates three files—ref.cpp (SystemC reference model), test.cpp (random stimulus generator), main.cpp (co-simulation harness)—compiled via Verilator alongside the DUT. Cycle-by-cycle output comparison produces an error trace with logged states, inputs, and register values.
  4. Judger/Fixer loop: On functional mismatch, Judger analyzes the error trace to determine whether fault is in the reference model or DUT, then Fixer patches the JSON IR (not the Verilog directly), and Coder regenerates Verilog from corrected IR.

SKT-FSM benchmark: 67 FSM samples from 100 open-source FSMs, complexity-scored via weighted composite of normalized state count (0.5), edge count (0.3 weight on edges actually weighted 0.5 in formula), and line count; split into Easy/Medium/Hard tiers.

Key Contributions

  • SKT-FSM: First hierarchical FSM benchmark with 67 samples across three complexity tiers (Easy: 27, Medium: 27, Hard: 13), including design descriptions, reference models, and test programs.
  • JSON IR + toolchain: JSON intermediate representation with a json2yaml converter and extended fsm2sv backend (customizable clock/reset names, internal registers beyond state register) to eliminate LLM-direct Verilog syntax errors.
  • SystemC auto-testbench generation: First framework to generate SystemC reference models and co-simulation harnesses automatically, eliminating reliance on pre-written benchmark test programs.
  • Judger-Fixer diagnostic loop: Error traces with cycle-level state logs enable root-cause localization (DUT vs. reference model) before targeted repair.

Results

  • AutoFSM (DeepSeek-V3) vs. MAGE (DeepSeek-V3): pass@1 58.21% vs. 46.27% (+11.94%); syntax error rate 0.75% vs. 1.24% (−0.49%).
  • AutoFSM (GPT-4o) vs. MAGE (GPT-4o): pass@1 44.78% vs. 38.81% (+5.97%); syntax error rate 5.22% vs. 22.84% (−17.62%).
  • AutoFSM (GPT-4o) vs. Qwen2.5-Max (standalone): 44.78% vs. 41.79% (+2.99%).
  • Hard tasks (DeepSeek-V3): AutoFSM 46.15% vs. MAGE 15.38% (+30.77%).
  • Ablation: Removing Tester drops pass@1 from 58.21% → 46.27% (−11.94%); further removing JSON IR drops to 50.75% with syntax error rate rising to 11.94%; bare DeepSeek-V3 yields 35.80% at 14.93% syntax error rate.

Limitations

  • Evaluated only on FSM generation; generalization to broader RTL constructs (e.g., pipelines, arbiters) is not demonstrated.
  • SKT-FSM has only 67 samples, limiting statistical robustness, particularly for the Hard tier (13 samples).
  • SystemC reference model is also LLM-generated and can itself be incorrect; the Judger must distinguish DUT vs. reference faults, which may fail on ambiguous traces.
  • Maximum iteration cap of 20 agent actions per round may leave hard cases unresolved without further tuning.
  • No support for complex nested or hierarchical state machines (stated future work).
  • Benchmark construction uses GPT-4-generated design descriptions with manual review—potential for LLM-introduced description artifacts.

Relevance to Harnesses / Meta-Harnesses

AutoFSM is a concrete domain-specific meta-harness: it wraps a backend tool (fsm2sv/Verilator/iverilog) with a multi-agent orchestration layer (MetaGPT) that manages IR generation, verification, compilation, co-simulation, fault diagnosis, and repair as a deterministic pipeline with feedback loops. The pattern of using an IR as a stable intermediate artifact to decouple LLM reasoning from brittle syntax generation is directly applicable to any harness that drives LLMs to produce structured tool inputs. The Judger/Fixer loop—where a diagnostic agent analyzes execution traces to route repairs to the correct upstream artifact—is a reusable meta-harness design pattern for iterative code generation harnesses. This paper demonstrates how specialized toolchain integration (SystemC, fsm2sv) within a multi-agent harness can replace the need for human-authored golden references, a key scalability challenge for harness designers.