Skip to content

Multi-Docker-Eval: A `Shovel of the Gold Rush' Benchmark on Automatic Environment Building for Software Engineering

🕒 Published (v1): 2025-12-07 16:43 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Multi-Docker-Eval is the first multilingual benchmark for evaluating LLM agents on automated Docker-based environment configuration for repository-level software engineering tasks. It covers 334 issues across 40 repositories in 9 programming languages, measuring both correctness (Fail-to-Pass rate) and resource efficiency. Current best models achieve only 37.7% F2P, with Docker dependency resolution as the dominant failure mode.

Problem

Scaling automated software engineering (SWE) pipelines requires reliable, reproducible execution environments—yet no benchmark evaluates this "shovel" step rigorously. Prior work (EnvBench, INSTALLAMATIC-bench) is language-specific (Python or JVM only), measures only setup success without validating test correctness, and ignores efficiency metrics like wall time, token cost, and Docker image size.

Method

The benchmark defines a task as: given a triplet \(\langle R, D, P^* \rangle\) (repository, natural-language issue description, ground-truth patch), an agent must produce (1) a runnable Dockerfile and (2) a test script \(T(\cdot)\) such that \(T(R)\) fails and \(T(R \oplus P^*)\) passes. Repositories are filtered to 1000–1500 GitHub stars, ≥20 forks, ≥10 contributors, ≤100 MB size, with pull requests created after July 31 2025. Difficulty is labeled "Easy" (trivially installable via e.g. pip install -r requirements.txt, 20.1%) vs. "Hard" (79.9%). Two agent frameworks are evaluated: SWE-Builder (4-agent multi-agent with repair loop and memory pool) and RepoLaunch (single-agent sequential bash interaction). Ten LLMs are tested: 7 open-source (DeepSeek-v3.1/R1, GPT-OSS-20/120B, Kimi-K2-0905/thinking, Qwen3-235B-A22B) and 3 closed-source (Claude-Sonnet-4, GPT-5-Mini, Gemini-2.5-Flash). Primary metric is Fail-to-Pass rate (F2P); process metrics include token consumption, wall time, CPU seconds, Max RSS, and average Docker image size.

Key Contributions

  • First multilingual, multi-dimensional benchmark for automated executable environment configuration, spanning 9 languages and 334 real-world issues.
  • Comprehensive evaluation of 10 LLMs across two agent frameworks with both outcome and process metrics.
  • Empirical framework comparison (SWE-Builder vs. RepoLaunch) with quantified architectural trade-offs.
  • Bottleneck analysis isolating Docker build failure as the primary limiting factor, distinct from test-script generation.
  • Design guidelines for scalable, cost-effective automated SWE pipelines.

Results

  • Best F2P: 37.72% (DeepSeek-v3.1 on SWE-Builder); closed-source best is Claude-Sonnet-4 at 35.53%.
  • Model size and reasoning length are not decisive: Kimi-K2-0905 matches top F2P (37.62%) using ~120K tokens vs. GPT-5-Mini's ~443K tokens for 34.13% F2P.
  • Framework gap is large: SWE-Builder averages 30.58% F2P vs. RepoLaunch at 8.85% F2P (≈3.5× difference); RepoLaunch uses 3.5× more input tokens (504K vs. 166K) but generates 4.2× fewer output tokens.
  • Commit rate overestimates success: Claude-Sonnet-4 commits 47.41% of runs but achieves only 35.53% F2P; Qwen3-235B-A22B shows similar overconfidence.
  • Language variation: Go achieves 54.5% F2P (commit 64.25%); Java is worst at 10.8% F2P; C/C++ and Rust also underperform due to system-level dependency complexity.
  • Failure mode breakdown (SWE-Builder average): Docker-build failures 36.09%, missing/non-executable scripts 18.12%, false passes 12.70%, post-submission env dependency failures 2.63%.
  • Thinking-enhanced models (DeepSeek-R1, GPT-5-Mini, Kimi-K2-thinking) reduce script-generation failures but do not improve Docker build success, confirming chain-of-thought aids code synthesis but not system-level dependency resolution.

Limitations

  • Benchmark is static (repositories sampled at a fixed cutoff), so data contamination risk increases over time.
  • Only two agent frameworks evaluated; results may not generalize to other architectures.
  • "Easy" subset is small (67/334 instances, 20.1%), limiting statistical power for difficulty-stratified analysis.
  • Metrics do not capture partial environment correctness (e.g., a Docker build that installs 90% of dependencies correctly).
  • No human-expert baseline is reported, making it unclear how close automated agents are to human-level configuration.
  • F2P ceiling may reflect benchmark difficulty rather than agent capability limits, as test-script quality is not independently assessed.

Relevance to Harnesses / Meta-Harnesses

Multi-Docker-Eval directly benchmarks the environment-provisioning layer that underlies all SWE harnesses: the automated construction of reproducible Docker environments and test scaffolding is itself a meta-harness task. The paper's comparison of SWE-Builder (multi-agent, repair-loop, memory pool) vs. RepoLaunch (single-agent sequential) operationalizes key harness design choices—agent decomposition, iterative self-correction, and validated-configuration reuse—and quantifies their impact on reliability and resource efficiency. The finding that multi-agent collaboration with feedback loops yields 3.5× higher F2P than single-agent sequential execution is direct evidence for the value of hierarchical, memory-augmented harness architectures. For researchers building or evaluating meta-harnesses, the benchmark's process metrics (token budget, wall time, Docker size) provide a practical cost model for capacity planning at scale.