Skip to content

Process-Level Trajectory Evaluation for Environment Configuration in Software Engineering Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR; process-level trajectory evaluation for SE agent environment configuration tasks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EnConda-Bench is a process-level trajectory evaluation benchmark for software engineering agents performing environment configuration. Rather than measuring only end-to-end build success, it decomposes agent capability into planning, perception (error localization), feedback (repair reasoning), and action (fix execution) by injecting realistic errors into repository READMEs and evaluating agents against gold-standard JSON annotations in Docker. Key finding: agents can identify error types but fail to convert that understanding into effective corrective actions.

Problem

Existing environment configuration benchmarks (INSTALLAMATIC, EXECUTIONAGENT, EnvBench, SetupBench) report only aggregate end-to-end build/test success, making it impossible to pinpoint which capability stage—planning, error localization, repair reasoning, or fix execution—causes failure. Additionally, high-quality, correctly buildable repositories are scarce, creating a data construction bottleneck that limits benchmark scale.

Method

Data construction pipeline: 1. Repository selection: GitHub repos filtered to ≥10 stars, ≥1,000 commits, ≥10 closed issues; supplemented by repos from prior benchmarks with human-verified setups. 2. Error synthesis: claude-4-sonnet and gemini-2.5-pro are prompted to inject exactly two errors per README from six canonical categories (Dependency Installation Error, Command Usage/Syntax Error, File Path/Missing File Error, Logical Order Error, Version Compatibility Error, Other). Each injection produces a structured JSON with error type, description, candidate fixes, and ground-truth fix. 3. Automatic Docker validation: gpt-4.1-mini (intentionally weaker to avoid implicit auto-fix) generates a shell script from the erroneous README; if setup fails and is repaired by the gold fix, the error is valid. 4. LLM + human filtering: gpt-4.1-mini screens for error validity, README explicitness, and fix correctness; human evaluators review survivors (LLM–human agreement: 98.5%). Valid instances are split/merged to vary error count (1–10+), yielding 4,201 READMEs and 9,471 errors from 323 repositories.

Evaluation suite: - Process-level: Precision/Recall/F1 on predicted error types; GPT-4.1-mini judges description accuracy and fix accuracy against gold JSON. - End-to-end: Agent-generated shell script runs inside pinned-commit Docker container; pass requires successful build, correct test execution, and clean exit (Pass@1).

Key Contributions

  • First benchmark providing process-level trajectory assessment for environment configuration in SWE agents, covering planning, perception, feedback, and action.
  • Automated data construction pipeline scaling from 323 repos to 4,201 task instances with minimal manual effort; synthetic errors are statistically indistinguishable from real ones (54.7% vs. 58.0% expert misclassification rate).
  • Evaluation of zero-shot LLMs, code agents (OpenHands, SWE-Agent), and environment-setup agents (INSTALLAMATIC, Repo2Run) revealing specific capability gaps at each trajectory stage.
  • Dataset and code released at https://github.com/TencentYoutuResearch/EnConda-Bench.

Results

  • Zero-shot LLMs show high recall but low precision for error typing (GPT-4.1: Rec 90.6, Pre 33.4, F1 48.8); poor fix suggestions and weak end-to-end performance.
  • Code agents improve error perception (OpenHands + DeepSeek-V3: F1 58.7, description acc 51.9) but action remains limited (fix acc 33.8, Pass@1 9.1).
  • Environment-setup agents achieve the best overall scores: Repo2Run + Claude-4 reaches F1 60.6, description acc 52.2, fix acc 47.3, Pass@1 22.9.
  • All systems show a persistent gap between error description accuracy and fix accuracy, confirming that translating diagnosis into corrective action is the primary bottleneck.
  • EnConda-Bench difficulty (mean score 3.95/5) closely matches real-world benchmarks (EnvBench: 4.08, EXECUTIONAGENT: 3.85), validating realistic applicability.

Limitations

  • Errors are LLM-synthesized; despite strong realism validation, edge-case distributions from real-world repositories may not be fully captured.
  • Pass@1 remains low across all systems (best: 22.9%), suggesting the benchmark is quite hard but leaving headroom; it is unclear how well findings generalize beyond Docker-based Python/multi-language repos.
  • The GPT-4.1-mini judge for description/fix accuracy introduces an evaluation model dependency and potential consistency variance.
  • Paper text is truncated; additional ablations or error-category breakdowns may not be fully reported here.

Relevance to Harnesses / Meta-Harnesses

EnConda-Bench's data construction pipeline is itself a multi-stage meta-harness: it orchestrates heterogeneous LLM calls (Claude + Gemini for error synthesis, GPT-4.1-mini for validation and filtering), Docker execution environments, and automated consistency checks to produce evaluation-ready instances at scale—a pattern directly analogous to paper-digest or test-harness meta-pipelines. The evaluation suite acts as a harness wrapper around arbitrary agent frameworks, extracting structured signals from free-form trajectories and feeding them into standardized metric computation. For researchers building harnesses, this paper demonstrates how to design process-level instrumentation (trajectory segmentation → capability-specific scoring) that avoids over-reliance on aggregate end-to-end metrics—a key architectural principle for evaluating harness-driven agentic pipelines.