Skip to content

Workflows vs Agents for Code Translation

🕒 Published (v1): 2025-12-15 20:35 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper empirically compares a fixed expert-designed workflow against an MCP-based agentic framework for the syntax-repair stage of a MATLAB-to-HDL transpilation pipeline. The agentic approach consistently outperforms the baseline at small and mid scales by dynamically invoking tools only when needed. The central finding is that how tools are invoked (conditionally vs. unconditionally) matters more than which tools are available.

Problem

LLMs have limited HDL training data, making direct end-to-end MATLAB→VHDL translation brittle; most failures occur in the syntax-repair stage. Expert-designed fixed-script repair loops are brittle under diverse error profiles, while naive RAG inclusion actively degrades performance by cluttering the model's context with irrelevant code.

Method

Two syntax-repair strategies are integrated into a MATLAB→HDL pipeline and compared across 42 signal-processing functions:

  1. Expert-designed (non-MCP) baseline: A fixed loop with a large expert-written prompt; uses GHDL compiler feedback iteratively (up to T=10 iterations, K=3 candidates per run, R=12 trials).
  2. Agentic MCP flow: The LLM receives a minimal prompt and a menu of three MCP tools — (a) GHDL syntax check, (b) RAG retrieval (top-3 from >1000 VHDL functions via Infinity embeddings + FAISS), and (c) a code-rewrite tool that delegates generation to a clean-context sub-agent. Context is aggressively pruned after each failed attempt (only a brief summary carried over) to avoid contamination. RAG is invoked conditionally: only when no progress is made or compiler errors indicate missing idioms.

Models tested: Qwen3-8B, Qwen3-30B-A3B, Qwen3-235B-A22B (no fine-tuning).

A third ablation variant — Non-MCP+RAG — unconditionally appends top-3 retrieved examples to every expert-flow prompt, isolating the effect of conditional vs. naive tool use.

Key Contributions

  • Empirical comparison of fixed workflow vs. MCP-based agentic flow across three model scales on a real MATLAB→VHDL transpilation task.
  • Demonstration that conditional tool invocation (not tool availability) is the primary driver of improvement; naive RAG inclusion is actively harmful at 8B and 30B.
  • Practical design guidelines: minimal prompts, aggressive context resets, separation of planning from generation (planner produces instruction list → clean-context generator produces VHDL).
  • Documentation of a scale-sensitive interaction: agentic frameworks are most impactful at mid-scale (30B), where they compensate for capacity limits without being redundant.

Results

Qwen3-30B (largest gains): - Candidate-level syntax pass: 51.9% → 75.0% (+23.1 pp) with MCP - Function-level syntax pass: 81.2% → 92.3% (+11.1 pp) - Simulation reach rate: 72.1% → 95.3% (+23.2 pp) - Final end-to-end success: 33.5% → 42.1% (+8.6 pp) - Non-MCP+RAG (naive) collapses simulation reach to 44.0% and final success to 19.5%

Qwen3-8B: - Function-level syntax pass: 76.7% → 90.7% (+14 pp) - Simulation reach: +>20 pp - Final success: 18.3% → 23.2% (+4.9 pp); Non-MCP+RAG has negligible effect vs. baseline

Qwen3-235B: - Baseline already achieves 93% function-level syntax pass; MCP pushes to 100% - Final success: Non-MCP 53.5% → MCP 55.8% (+2.3 pp); Non-MCP+RAG achieves highest at 58.1% (naive RAG acts as few-shot priming at large scale)

Limitations

  • Dataset is 42 functions from a single domain (signal processing); generalization to other error profiles is untested.
  • Downstream semantic bottleneck persists: improved syntax repair does not fully propagate to end-to-end success.
  • Dataset is internal and not publicly released (a cleaned subset is planned).
  • The context-saturation mechanism hypothesized to explain scale effects is not directly ablated (no length-controlled experiments with equalized token budgets).
  • 235B result for Non-MCP+RAG vs. MCP (+2.3 pp) may fall within run-to-run variance (σ ≈ 2–3 pp); no confidence intervals reported.
  • Compute requirements for 235B are not logged, limiting reproducibility.

Relevance to Agentic AI / LLM Agents

This paper provides a controlled, quantitative case study of a core agentic design question: when does autonomous tool selection outperform a fixed expert workflow? The finding that conditional invocation — not tool availability — drives gains is directly actionable for anyone building agentic coding systems, and the "naive RAG hurts small models, helps large models" result is a concrete, replicable design heuristic. The planning/generation separation pattern (clean-context sub-agent for generation) is a practical instantiation of the broader principle of modular context management in multi-step agentic pipelines. The scale-dependence of agentic benefit connects to ongoing work on when scaffolding adds value vs. when model capacity makes it redundant — a key open question for deploying LLM agents at different resource tiers.