Skip to content

EnvBench: A Benchmark for Automated Environment Setup

🕒 Published (v1): 2025-03-18 17:19 UTC · Source: Arxiv · Venue: ICLR 25 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EnvBench is a benchmark of 994 Python and JVM repositories for the task of automated environment setup—generating shell scripts that correctly configure a development environment from scratch. Existing benchmarks were too small (≤50 repos) to capture real configuration diversity. The best evaluated agent (ReAct Bash Agent + GPT-4o) succeeds on only 29.47% of JVM and 6.69% of Python repositories, confirming the benchmark remains highly challenging.

Problem

Prior environment setup benchmarks (INSTALLAMATICbench: 40 repos; EXECUTIONAGENTbench: 10 repos per language) are too small and biased toward popular, well-documented projects (≥1000 stars), preventing rigorous evaluation of the full range of configuration challenges. Evaluation metrics in prior work also relied on test-suite execution, which requires manual effort and limits scalability. No reproducible, large-scale, automatically evaluable benchmark existed for this task.

Method

Benchmark construction: GitHub repositories are filtered by language (Python, Java, Kotlin), license, and activity signals (≥1000 commits, ≥10 issues/contributors/stars, last commit ≥ Jan 2024), yielding 2,590 Python and 1,688 JVM candidates. Repositories are filtered down by requiring a single recognized dependency manager (pip/Poetry for Python; Gradle/Maven for JVM), excluding monorepos and Docker-heavy repos. A deterministic baseline script is applied and repositories it can fully configure (15.9% Python, 31.6% JVM) are excluded to enforce non-triviality. Final dataset: 329 Python + 665 JVM repositories.

Evaluation: Each approach receives the full repository contents and must emit a shell script. Execution is sandboxed in Docker containers with predefined base images. Correctness is measured by: (1) Python: pyright reportMissingImports count (zero = pass); (2) JVM: Gradle/Maven compilation exit code (zero errors = pass). Metrics: binary pass@1 and continuous avgErrs.

Baselines evaluated: - Zero-shot LLM: single prompt with directory structure, README, config files → shell script. - Installamatic Agent: two-phase agent (repository exploration → single-shot script generation), adapted from Milliken et al. (2024). - Bash Agent: ReAct agent with a single execute_bash tool, iteratively building the script inside a Docker sandbox; analogous to a simplified ExecutionAgent (Bouzenia & Pradel, 2024).

Both GPT-4o and GPT-4o-mini are tested as backbones.

Key Contributions

  • A benchmark of 994 repositories (329 Python, 665 JVM) with non-trivial configuration challenges, 13–66× larger than prior work.
  • Two automated, scalable evaluation metrics (static import analysis via pyright; compilation check via Gradle/Maven) that avoid manual test-suite inspection.
  • A Dockerized evaluation suite ensuring reproducibility and host-system safety.
  • Empirical evaluation of three baseline strategies × two LLMs, establishing performance floors and identifying failure modes.
  • Public release of benchmark, evaluation suite, and full experiment trajectories.

Results

  • Best overall: Bash Agent + GPT-4o — 29.47% pass@1 (JVM), 6.69% pass@1 (Python).
  • Zero-shot LLM + GPT-4o-mini: 11.13% (JVM), 5.47% (Python) — substantially worse on JVM.
  • Installamatic Agent: 3.01% (JVM, GPT-4o-mini), 4.86% (Python, GPT-4o) — worst baseline, likely due to prompts tuned for high-star repos with rich documentation.
  • Human expert upper bound (30-repo sample): 66.7% pass@1 vs. 10.0% for best baseline — large gap.
  • For Python avgErrs: Bash Agent + GPT-4o averages 52.00 missing imports per repo (over repos where script exited cleanly).
  • Relative to the deterministic script on Python: 31–53% of repos see fewer missing imports with LLM-based approaches; 13–24% see more, with average degradation ≥200% in those cases.
  • LLM approaches without explicit error feedback frequently produce scripts with non-zero exit codes, confirming that iterative repair is essential.

Limitations

  • Evaluation covers only Python and JVM (Java/Kotlin); C, C++, JavaScript, and others are excluded.
  • Metrics cover the build/install step only, not runtime or test-suite correctness — some failures invisible to the metrics would surface at test time.
  • Only proprietary models (GPT-4o, GPT-4o-mini) are evaluated; open-weight models are untested.
  • Repositories requiring Docker internally are excluded, creating a gap for containerized projects.
  • avgErrs is only computable over the subset of repos where the generated script exits cleanly, making cross-baseline comparison on that metric uneven.
  • No regeneration attempts are allowed for Installamatic in the reported setting, potentially underestimating its ceiling.

Relevance to Harnesses / Meta-Harnesses

EnvBench directly addresses the prerequisite infrastructure problem for any execution-based harness: before a meta-harness can run tests, reproduce experiments, or evaluate code agents, the environment must be correctly configured. The paper's Docker-sandboxed evaluation suite is itself a harness pattern—it orchestrates clone → script execution → metric check per repository, which mirrors the structure of meta-harnesses that pipeline agent outputs through automated verification. The finding that agentic, iterative approaches (ReAct + bash tool) substantially outperform single-shot generation is directly actionable for harness designers choosing how to wire environment-setup agents into larger pipelines. The scalable, automated metrics also demonstrate a key design principle for meta-harnesses: replacing expensive manual checks with lightweight proxies (static analysis, compilation) that are good enough to guide agent improvement at scale.