Skip to content

RAT: RunAnyThing via Fully Automated Environment Configuration

🕒 Published (v1): 2026-04-25 07:45 UTC · Source: Arxiv · link

Why this paper was selected

Automated repo environment configuration; foundational infrastructure for coding agent harnesses

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RAT (RunAnyThing) is an LLM-driven, modular multi-stage agent framework that fully automates repository environment configuration across multiple programming languages without relying on pre-existing Dockerfiles or CI artifacts. It pairs a language-aware abstraction layer, semantic Docker image retrieval, a specialized toolset, and long-term expertise accumulation within containerized sandboxes. The authors also release RATBench, a 2,500+ repo multilingual benchmark with execution-verified difficulty tiers, on which RAT achieves a 36.1% average ESSR improvement over strong baselines.

Problem

Autonomous code agents require runnable environments to verify generated code, yet environment configuration is a labor-intensive bottleneck. Prior approaches (INSTALLAMATIC, ExecutionAgent, Repo2Run) assume pre-existing Dockerfiles, CI logs, or installation scripts and are largely limited to Python or Java, making them ill-suited for scaling to thousands of diverse, uncurated, multilingual repositories. Existing benchmarks compound this by relying on static metrics, small manually curated samples, or single-language coverage, preventing rigorous evaluation.

Method

RAT executes a four-component multi-stage pipeline inside a containerized sandbox:

  1. Language-Aware Abstraction — identifies the dominant language by code proportion and encapsulates language-specific heuristics (dependency manifests like pom.xml/Cargo.toml, package-manager protocols, test runners) into a unified interface, decoupling language logic from the general agent loop.

  2. ImageRetriever — performs LLM-based semantic analysis of repository docs and configuration artifacts to infer runtime requirements (language versions, OS variant, DL framework), retrieves candidate images from a predefined standard pool (e.g., python:3.10, openjdk:17) and Docker Hub, then applies an LLM scoring mechanism to select the optimal base image.

  3. Configuration Modes — Standard mode follows a fixed deterministic workflow per language; Automated mode uses a ReAct (Thought → Action → Observation) loop for adaptive, repository-specific discovery. Mode selection depends on repository characteristics.

  4. Specialized Configuration Toolset + Robust Sandbox — high-level tools (Read File, Edit File, View Outline, Error Recovery, Change Version, Test Synthesis, CI/CD Config, Issue Retrieval) abstract raw terminal commands into configuration-aware interfaces. A template-based Docker generation module builds a tailored Dockerfile with pre-flight validation before agent deployment.

  5. Long-term Expertise Accumulation — serializes lessons from historical execution trajectories into a structured JSON schema to improve future configuration sessions.

The evaluation metric is the Environment Setup Success Rate \(\text{ESSR} = N_{\text{pass}} / N_{\text{verified}}\), assessed under three settings: S1 (artifacts + tests present), S2 (tests present, no containerization scripts), S3 (neither tests nor scripts).

Key Contributions

  • RAT framework: modular, language-extensible multi-stage pipeline for automated environment configuration across Python, Java, Rust, JS/TS, and Go.
  • RATBench: 2,500+ GitHub repositories with stratified sampling over project size and popularity, 5 languages, automated collection, execution-verified validity, and explicit difficulty tiers — the first benchmark with all these properties simultaneously.
  • ESSR metric with three inference-difficulty scenarios (S1–S3) that isolate configuration failures from pre-existing code defects.
  • Demonstrated that RAT surpasses senior human engineers in ESSR (91.52% vs. 89.41%) while requiring no human intervention and supporting parallelized large-scale deployment.
  • Scaling-law behavior: ESSR improves monotonically with increased execution steps.

Results

  • Python (DeepSeek-V3): RAT 63.2% vs. Repo2Run 44.8% vs. SWE-agent 15.5% vs. pipreqs 35.8%.
  • Multilingual (DeepSeek-V3): Java 41.3%, Rust 98.7%, JS/TS 68.7%, Go 71.7% — baselines are mostly inapplicable or near zero.
  • Average improvement over SWE-agent: +36.1% across all languages.
  • Scenario breakdown (Python, DeepSeek-V3): RAT avg 70.7% vs. Repo2Run avg 22.7%; in S3 (most underspecified): RAT 92.0% vs. Repo2Run 3.0%.
  • Backbone scaling (Python): RAT with GPT-5.2 reaches 86.8% ESSR vs. Repo2Run's 25.1%.
  • vs. human engineers: RAT ESSR 91.52% vs. 89.41%, at the cost of 31.26 min vs. 13.73 min latency per repo.
  • Ablation: removing ImageRetriever drops ESSR to 40.5% (−22.7 pp); removing the toolset drops to 55.7% (−7.5 pp) while more than doubling token usage and latency.

Limitations

  • RAT's latency (31.26 min/repo) is 2.25× that of human engineers; token cost is substantial (~421.9K tokens/repo with DeepSeek-V3).
  • Automated mode yields a modest ESSR decrease versus standard mode, trading accuracy for speed and token efficiency.
  • Benchmark covers only 5 languages; behavior on niche or domain-specific toolchains (e.g., Fortran, R, MATLAB) is untested.
  • ESSR for S1 (artifact-guided) is still only 50.5%, meaning even with existing Dockerfiles RAT fails roughly half the time.
  • Comparison against human engineers is on a small, non-representative sample of three repositories.

Relevance to Harnesses / Meta-Harnesses

RAT is a direct instance of an automated meta-harness: it wraps arbitrary repositories in reproducible, executable environments — the prerequisite layer that all downstream benchmark harnesses, RL reward loops, and data-synthesis pipelines depend on. The three roles the authors identify for automated env config (scalable data synthesis, execution-based reinforcement, CI/CD reproducibility) map precisely to the infrastructure concerns that meta-harness designers face when scaling evaluation across codebases. The Long-term Expertise Accumulation module, which serializes cross-session knowledge into structured JSON, is an explicit meta-harness memory mechanism. RATBench itself demonstrates how a harness can be bootstrapped without human curation by applying stratified sampling and execution-based verification — a reusable pattern for anyone building large-scale automated evaluation pipelines.