Skip to content

BabelCoder: Agentic Code Translation with Specification Alignment

🕒 Published (v1): 2025-12-07 15:57 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

BabelCoder is a multi-agent framework for automated code translation that decomposes the task into three specialized agents—Translation, Test, and Refinement—coordinated through structured inter-agent communication. The Refinement Agent generates and validates Natural Language Specifications (NL-Specifications) to bridge semantic gaps between source and target languages, then applies layered bug repair including Spectrum-Based Fault Localization (SBFL) and LLM-based scope estimation. Evaluated on four benchmarks across five languages, BabelCoder achieves 94.16% average accuracy, outperforming state-of-the-art baselines in 94% of cases by 0.5%–13.5%.

Problem

Existing LLM-based code translation approaches share three key deficiencies: (1) they rely on single-step, end-to-end translation that cannot adapt to complex or semantically ambiguous code; (2) refinement mechanisms are shallow—focused on syntactic or variable-level fixes rather than semantic correctness; and (3) multi-path methods (e.g., InterTrans) achieve better accuracy but at prohibitive computational cost due to excessive LLM invocations. No prior system integrates structured inter-agent collaboration with language-agnostic intermediate specifications to guide both translation and iterative repair.

Method

BabelCoder orchestrates three agents in a closed-loop pipeline:

  • Translation Agent: Generates an initial translation from source code or from a validated NL-Specification using a fixed prompt template. Delegates to the Refinement Agent before and after translation.
  • Refinement Agent: Operates in six sequential workflows:
  • NL-Specification Generation: Produces pseudocode-style, language-agnostic line-by-line specification from the source.
  • NL-Specification Validation: Aligns the NL-Spec against the source via LLM-based line-by-line alignment; revises and regenerates translation from the corrected spec.
  • NL-Specification Augmentation: A state machine selects between raw source code and NL-Spec as Translation Agent input, based on test-case pass rates across iterations.
  • Fixing Errors: Resolves compilation errors first, then runtime/assertion errors via LLM prompts augmented with summarized error messages.
  • Bug Localization: Uses SBFL (suspiciousness matrix from coverage + pass/fail data) and LLM-based scope estimation (categorizes fault into Input Processing, Output Formatting, Variable Declaration, Loop Blocks, or Conditional Blocks).
  • Fix Error within Scope: Invokes LLM with bug scope and suspicious lines to perform targeted repair.
  • Test Agent: Generates corner-case test inputs via LLM, executes source to derive oracles, runs translated code, collects coverage reports (language-specific tools), and summarizes filtered error messages for the Refinement Agent.

The underlying LLM is GPT-4o throughout.

Key Contributions

  • Multi-agent architecture with three specialized agents and explicit inter-agent communication protocols for iterative code translation and repair.
  • NL-Specification pipeline: generation, LLM-based validation/alignment, and state-machine-driven augmentation as a language-agnostic semantic bridge.
  • Dual bug localization: SBFL for line-level suspiciousness scoring + LLM-based scope estimation for category-level fault attribution.
  • Java port of EvalPlus test cases, released publicly to enable correctness evaluation of Java translation targets.

Results

  • Average accuracy of 94.16% across all benchmarks and language pairs.
  • Outperforms all four baselines (Lost in Translation, UniTrans, TransAgent, InterTrans) in 94% of evaluation cases, by margins of 0.5%–13.5%.
  • Evaluated on: AVATAR (479 data points, Java↔Python, 8 pairs), CodeNet (1,000 samples, 20 language pairs across C/C++/Go/Java/Python), EvalPlus (164 Python→Java problems, 2,681 test cases), TransCoder (464 Python problems with LLM-generated test oracles).

Limitations

  • Evaluated with a single LLM (GPT-4o); generalizability to smaller or open-source models is untested.
  • Scope is restricted to function-level and file-level translation; repository-level translation (with cross-file dependencies, architectural preservation, context-window constraints) is deferred to future work.
  • The state machine governing NL-Spec vs. source-code selection is heuristic (test pass rate); it may not converge efficiently on adversarial inputs.
  • Computational cost analysis (number of LLM invocations per sample) is presented for specific examples (e.g., 15 invocations for one EvalPlus problem) but no systematic cost comparison against baselines is reported in the provided text.
  • Security and code quality metrics (RQ2) are mentioned but not detailed in the provided excerpt.

Relevance to Harnesses / Meta-Harnesses

BabelCoder is a concrete instance of a task-specific multi-agent harness: a fixed orchestration graph where three specialized agents (Translation, Test, Refinement) communicate through structured artifacts (NL-Specifications, test reports, coverage matrices) under a coordination protocol. The state-machine-driven NL-Specification Augmentation step is particularly relevant as a lightweight meta-level controller that dynamically selects agent inputs based on observed runtime signals—analogous to feedback-driven routing in meta-harness designs. The layered repair pipeline (compile → runtime → SBFL → scope → fix) demonstrates how a harness can chain heterogeneous sub-strategies without a monolithic controller. Researchers building general-purpose meta-harnesses can study BabelCoder's inter-agent message passing and iterative feedback loop as a domain-specific reference implementation.