Skip to content

Workflows vs Agents for Code Translation

🕒 Published (v1): 2025-12-15 20:35 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper compares a fixed expert-designed workflow against an MCP-based agentic flow for the MATLAB-to-HDL syntax repair stage of a transpilation pipeline. The agentic approach outperforms the rigid workflow at small and mid-scale models (8B, 30B) by enforcing conditional tool use and aggressive context pruning. The primary finding is that how tools are invoked matters more than which tools are available.

Problem

LLMs have sparse HDL training data, making direct MATLAB-to-HDL transpilation brittle. Syntax repair is the most common failure point in structured pipelines, but naive automated repair risks silently altering semantics. The core gap is: given a fixed tool set (compiler, RAG), should repair be orchestrated by a deterministic expert script or by an autonomous agent selecting tools dynamically?

Method

A MATLAB-to-HDL pipeline isolates the syntax repair stage for controlled comparison. Two repair strategies are evaluated on 42 MATLAB signal-processing functions across three model scales (Qwen3-8B, 30B, 235B):

  1. Expert-Designed Flow (baseline): Large expert-written prompt with full repair guidance; fixed loop calling GHDL compiler until syntax passes or iteration limit reached.
  2. Agentic MCP Flow: Minimal prompt (broken code + repair goal + tool menu); agent selects from: GHDL syntax check, RAG retrieval (FAISS over 1000+ VHDL functions, top-3 neighbors, Infinity embeddings), and a Code Rewrite tool that delegates generation to a clean-context sub-agent. Context is aggressively reset after each attempt (only a brief summary carried over). Retrieval is conditional: invoked only on no-progress or GHDL errors signaling missing idioms.

A third variant (Non-MCP+RAG) unconditionally appends top-3 RAG results to every expert-flow prompt, serving as a negative control.

The design evolved through four stages, converging on separation of planning from generation: a planner agent produces a compact instruction list; a clean-context generator produces the final VHDL from those instructions, never seeing prior failed attempts.

Key Contributions

  • Empirical comparison of structured workflow vs. MCP-based agentic repair across three model scales on a real-world MATLAB→VHDL pipeline.
  • Demonstration that conditional tool invocation is the primary driver of improvement—naive RAG attachment actively degrades performance at 8B and 30B.
  • Design guidelines: minimal prompts, aggressive context management, and separation of planning from generation as scale-sensitive levers.
  • Characterization of scale-dependent returns: agentic frameworks are most impactful at mid-scale (30B), where model capacity is sufficient to exploit orchestration structure but insufficient to self-filter context noise.

Results

30B (largest gains): - Candidate-level syntax pass: 51.9% → 75.0% (+23.1 pp) - Function-level syntax pass: 81.2% → 92.3% (+11.1 pp) - Simulation reach rate: 72.1% → 95.3% (+23.2 pp) - Final success: 33.5% → 42.1% (+8.6 pp) - Non-MCP+RAG (naive): simulation reach 44.0%, final success 19.5% (below baseline)

8B: - Function-level syntax pass: 76.7% → 90.7% (+14 pp) - Simulation reach: 60.5% → 90.7% (+30.2 pp) - Final success: 18.3% → 23.2% (+4.9 pp)

235B: - Function-level syntax pass: 93.0% → 100% (+7 pp) - Simulation reach: 100% → 100% (saturated) - Final success: 53.5% → 55.8% (+2.3 pp) via MCP; Non-MCP+RAG attains 58.1% (highest at this scale) - Run variability: σ ≈ 2–3 pp on intermediate metrics

Limitations

  • Dataset is 42 functions from a single domain (signal processing), internal to Code Metal; not yet publicly released.
  • Semantic bottleneck persists downstream of syntax repair—improved syntax pass rate does not fully propagate to end-to-end semantic equivalence.
  • At 235B, naive RAG slightly outperforms MCP on final success; the context-filtering hypothesis is unverified (no length-controlled ablations or confidence intervals reported).
  • Runtime and hardware details for Qwen3-235B not logged, hampering reproducibility at that scale.
  • No formal semantic safeguards (differential testing, equivalence checking) evaluated alongside the repair stage.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct empirical study of the structured workflow vs. autonomous agent design axis that is central to meta-harness design. The finding that a meta-harness with conditional tool dispatch (MCP) outperforms a fixed-script harness—and that naively passing all context is actively harmful—provides concrete, quantified evidence for why harness orchestration logic matters as much as the underlying model. The four-stage design evolution (accumulating tool outputs → interleaved reasoning → line-level edits → planner/generator separation) is a practical roadmap for iterating toward a robust meta-harness architecture. The scale-dependent returns imply that harness complexity should be matched to model capacity: simpler orchestration may suffice for frontier models, while mid-scale models benefit most from disciplined context hygiene enforced by the harness.