Skip to content

SAFEdit: Does Multi-Agent Decomposition Resolve the Reliability Challenges of Instructed Code Editing?

🕒 Published (v1): 2026-04-28 15:04 UTC · Source: Arxiv · link

Why this paper was selected

Multi-agent decomposition evaluated for code editing reliability improvement

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SAFEdit is a multi-agent framework for instructed code editing that decomposes the task into a Planner, Editor, and Verifier agent orchestrated via CrewAI, with a Failure Abstraction Layer (FAL) converting raw test logs into structured diagnostic feedback for iterative refinement. Evaluated on 445 EditBench tasks across five natural languages, it achieves 68.6% Task Success Rate (TSR), beating the best single-model baseline (claude-sonnet-4 at 64.8%) by +3.8pp and a ReAct single-agent baseline by +8.6pp. The iterative refinement loop alone contributes +17.4pp to overall TSR.

Problem

On the EditBench benchmark, 39 of 40 evaluated models achieve TSR below 60% on instructed code editing — a task requiring precise modification of existing code while preserving invariants and satisfying executable test suites. Existing multi-agent approaches target code generation or full repository-level issue resolution, not localized, instruction-driven editing with real test-based verification. Raw test log feedback is noisy and verbose, impeding effective iterative repair.

Method

SAFEdit orchestrates three specialized agents via CrewAI:

  1. Planner Agent: Given the instruction and visible code (under one of three visibility variants — CODE ONLY, HIGHLIGHT, HIGHLIGHT+CURSOR), produces a structured, variant-specific edit plan with fields: Observed Elements, Edit Intent, Target Description, Required Changes, Constraints. Generates no code.
  2. Editor Agent: Consumes the edit plan and original code; applies changes literally and minimally, outputting only edited fragments.
  3. Verifier Agent: Executes the full test suite in a sandboxed environment (30s timeout, process-level isolation) against real unit tests.

On test failure, the Failure Abstraction Layer (FAL) transforms raw logs through three stages: (1) log parsing to extract failed test names, exception types, and expected/actual values; (2) deterministic pattern-matching to classify failures into 14 structured types (e.g., ASSERTION_MISMATCH, TYPE_ERROR); (3) an explanation generator producing a structured feedback block with [TEST], [TYPE], [DIAGNOSIS], [ACTION], [EXPECTED], [ACTUAL], [CONFIDENCE] fields. This structured feedback re-enters the Editor for up to three refinement iterations. Backbone: GPT-4.1 at temperature 0.1 for all agents. Failure root-cause classification uses a GPT-4.1-based classifier with Pydantic-validated output across four categories: Instruction Hallucination (IH), Implementation Gap (IG), Regression Error (RE), Context Misalignment (CM).

Key Contributions

  • Planner–Editor–Verifier decomposition specifically designed for instructed code editing, separating planning from code generation to avoid instruction hallucination.
  • Failure Abstraction Layer (FAL): three-stage pipeline (parse → classify → explain) converting raw test logs into structured, actionable feedback without additional LLM calls (stage 2 uses deterministic pattern matching).
  • Iterative refinement loop capped at 3 iterations, contributing +17.4pp to TSR over first-pass performance; eliminates regression errors observed in single-agent baselines.
  • Automated error taxonomy (IH/IG/RE/CM) enabling qualitative failure redistribution analysis beyond pass/fail rates.
  • Multi-language evaluation on 445 EditBench tasks across English, Polish, Spanish, Chinese, Russian, revealing stable context-effect signals absent in prior single-model evaluations.

Results

  • SAFEdit TSR: 68.6% averaged across 5 languages and 3 visibility variants (445 tasks, 1335 instances total).
  • vs. best single-model baseline (claude-sonnet-4, HIGHLIGHT variant): +3.8pp (64.8% → 68.6%).
  • vs. ReAct single-agent baseline (same GPT-4.1 backbone, same iteration budget): +8.6pp.
  • Iterative refinement loop contribution: +17.4pp over first-pass (first-try success rate).
  • Three performance tiers observed under HIGHLIGHT variant: SAFEdit (high-60s), frontier single-models and ReAct (55–65%), remaining EditBench leaderboard models (<55%).
  • Regression Error (RE) category eliminated in SAFEdit; failures shifted from IH toward IG compared to single-agent baselines.
  • Context degradation is monotonic across variants for SAFEdit and ReAct (less context → lower TSR), contrasting with model-dependent and inconsistent effects in original EditBench study.

Limitations

  • Evaluated solely on EditBench; no evaluation on SWE-bench or repository-level tasks.
  • Maximum 3 refinement iterations — diminishing returns and potential overfitting at higher iteration counts noted but not empirically measured here.
  • FAL stage 2 uses deterministic pattern matching covering 14 failure types; novel failure modes outside this taxonomy are unclassified.
  • Error taxonomy classification (stage 3 and root-cause) relies on GPT-4.1, introducing potential misclassification and LLM cost.
  • All agents share the same backbone (GPT-4.1); no ablation across different backbone models for each agent role.
  • EditBench covers only Python and JavaScript/React; generalization to other languages is untested.

Relevance to Harnesses / Meta-Harnesses

SAFEdit is a concrete instance of a task-specific agentic harness: it wraps a backbone LLM in a fixed orchestration scaffold (CrewAI) with role-specialized agents, a structured inter-agent communication protocol, and a deterministic feedback transformation layer (FAL). The FAL in particular is a harness-level component — it sits between the execution environment and the LLM loop, normalizing raw environment signals into structured inputs the model can act on reliably. The iterative refinement loop with a hard cap and structured feedback channels is a canonical harness pattern for execution-grounded agent loops. For researchers building meta-harnesses, SAFEdit demonstrates that separating planning from execution within the harness (rather than giving a single agent free rein) materially reduces hallucination and regression errors, offering a decomposition template applicable beyond code editing.