Skip to content

Dockerless: Environment-Free Program Verifier for Coding Agents

🕒 Published (v1): 2026-06-26 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Dockerless is an environment-free agentic patch verifier for coding agents that replaces Docker-based unit-test execution by dispatching parallel sub-agents to explore a repository and answer targeted verification questions. It achieves 81.0 AUC on SWE-bench Verified—14.3 points above the strongest open-source verifier—and enables a fully environment-free post-training pipeline (SFT + RL) that matches environment-based post-training performance.

Problem

Execution-based verification for SWE-style coding agents requires building per-repository Docker images with pinned dependencies and custom test runners. This is expensive, fails on private or legacy codebases lacking reproducible environments, and blocks scalable post-training (SFT trajectory filtering and RL reward computation). Existing environment-free verifiers score patches only on surface-level textual similarity without inspecting the codebase, which is insufficient for functional equivalence judgments in complex SWE tasks.

Method

Dockerless operates in two stages given an issue \(x\), a reference patch \(y_{\text{ref}}\), and a candidate patch \(y\):

  1. Question generation & parallel exploration. The model generates \(K \in \{2\text{–}4\}\) verification questions derived from the issue and reference patch (e.g., "Is the fix applied to both XAxis and YAxis?"). \(K\) sub-agents run in parallel, using read-only shell tools (find, grep, rg) to gather evidence-backed answers \(\{(Q_k, A_k)\}_{k=1}^K\) from the repository.

  2. Judgment. A shared backbone conditions on \((x, y_{\text{ref}}, y, \{(Q_k, A_k)\})\) and outputs a binary verdict token; the continuous score is: $\(r_\phi(x, y) = \frac{\exp(\ell_1)}{\exp(\ell_0) + \exp(\ell_1)}\)$ where \(\ell_0, \ell_1\) are the logits of tokens 0 and 1.

Training uses rejection sampling on 3.7K execution-labeled issues from SWE-Gym and Multi-SWE-RL: a teacher model generates question-answer-judge trajectories \(\tau\), and only trajectories whose predicted verdict \(\hat{r}\) matches the ground-truth \(r^\star\) are retained as \(\mathcal{D}_{\text{rej}}\). The backbone is fine-tuned end-to-end with standard next-token cross-entropy over the full trajectory. For RL, Dockerless rewards are inserted into GRPO with group-normalized advantages; each reward is averaged over \(M\) independent evaluations for stability.

Key Contributions

  • Dockerless verifier: First environment-free verifier that grounds correctness judgments in agentic repository exploration rather than surface-level patch comparison.
  • Fully environment-free post-training pipeline: SFT trajectory filtering and RL reward computation both run on a minimal base image, eliminating per-repository Docker setup.
  • Rejection-sampling training: A teacher-model pipeline that generates and filters question-answer-judge trajectories against execution ground truth to train the verifier backbone.
  • Latency characterization: Shows that agentic verification overhead (180s) is only 7.2% of total RL per-rollout time, dominated by agent rollout generation (2308s).

Results

  • Verifier AUC: Dockerless reaches 81.0 on SWE-bench Verified and 72.1 on Multi-SWE-bench Flash, vs. DeepSWE Verifier (best open-source trained baseline) at 66.7 / 62.9 — a +14.3 / +9.2 point improvement; vs. GPT-5.4 (best frontier LLM judge) at 75.9 / 59.5 — a +5.1 / +8.2 point improvement.
  • SFT filtering: Dockerless 4K (top-ranked 4K of 16K env-free rollouts) achieves 60.6 / 47.7 / 35.3 on SWE-bench Verified / Multilingual / Pro, matching Env-based 4K (60.0 / 48.3 / 33.9) and substantially outperforming Random 4K (58.2 / 44.3 / 32.0). Training on all 16K unfiltered rollouts hurts vs. the base model.
  • End-to-end Dockerless-RL-9B: 62.0 / 50.0 / 35.2, improving over Qwen3.5-9B baseline by +2.4 / +8.7 / +2.9 and over the next-best open-source specialist SWE-Lego-8B by +20.8 / +31.0 / +19.1.
  • Env-free RL vs. env-based RL: Dockerless-RL-9B (env-free) achieves 62.0 / 50.0 / 35.2 vs. Test-Execution RL (env-based oracle) at 62.4 / 51.3 / 35.7 — within ~1 point gap.
  • Verification question ablation: AUC rises from 78.3 (K=0) to 81.0 (K=4); performance fluctuates beyond K=4, establishing 2–4 as the sweet spot.

Limitations

  • Requires a reference patch \(y_{\text{ref}}\) at verification time; not applicable to settings where no reference solution exists.
  • Read-only exploration: Sub-agents use only find/grep/rg; they cannot execute code, build the project, or run partial tests, so dynamic behaviors invisible to static analysis remain opaque.
  • Performance ceiling: Env-free RL still trails env-based oracle by ~0.4–1.3 points across benchmarks; does not fully close the gap.
  • Training data scale: Rejection-sampled from only 3.7K labeled issues; coverage of diverse repository types (non-Python, build-heavy, legacy) is uncharacterized.
  • Reward noise at K > 4: Extra verification questions introduce redundant or noisy evidence, suggesting the question generation module lacks a quality gate.

Relevance to Agentic AI / LLM Agents

Dockerless directly addresses the scalability bottleneck in training coding agents: by replacing Docker-based test execution with an agentic verifier, it decouples agent post-training from environment infrastructure, enabling RL and SFT on the long tail of real-world repositories. The architecture—question generation followed by parallel evidence-gathering sub-agents—is a concrete instantiation of multi-agent verification, showing that agentic decomposition can substitute for executable environments in reward modeling. This connects to broader work on LLM-as-judge and process reward models, but uniquely grounds judgments in live repository state rather than static context. For researchers tracking coding agents, Dockerless establishes environment-free post-training as a practical alternative to execution-based pipelines, with implications for scaling RL on private or enterprise codebases.