Skip to content

SETA: Scaling Environments for Terminal Agents

🕒 Published (v1): 2026-07-12 19:40 UTC · Source: Arxiv · link

Why this paper was selected

Scales terminal agent environments; practical benchmark infrastructure for agentic eval harnesses

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SETA is a scalable framework for generating verifiable terminal-agent training environments for reinforcement learning. It combines SETA-Synth (converting diverse human-validated sources into Docker-containerized, test-verified environments) with SETA-Evol (adaptively reshaping difficulty and diversity based on a training model's capability frontier). The resulting SETA-Env dataset (4,567 environments) trains Qwen3-8B to 12% pass@1 on Terminal-Bench 2.0, the best reported result for an 8B RL-trained model.

Problem

Terminal-agent RL training lacks large-scale, verifiable, and difficulty-controlled training data. Unlike repository-level coding tasks (SWE-bench), terminal workflows have no natural grounding signals (no PRs or issues), require simultaneous synthesis of executable environments, task instructions, and verification logic, and existing synthesis pipelines produce difficulty distributions skewed by source characteristics rather than calibrated to the training model's capability frontier.

Method

SETA consists of two pipelines sharing a unified verification mechanism, operating within the Harbor task format:

SETA-Synth converts five structured sources (Stack Overflow, Unix/Linux StackExchange, Kaggle notebooks, NL2Bash) into verifiable environments via a two-agent cascade: - Idea Agent: applies a shared base prompt plus a per-source adapter prompt to produce a source-agnostic draft specification; may invoke online search to fill missing context. - Datapoint Agent: instantiates the draft into a complete environment: Dockerfile, solve.sh, test_state.py, test.sh, instruction.md, and task.toml.

Verification requires (1) a no-op test (zero tests pass before any agent action) and (2) an oracle test (ground-truth solution passes all tests). A Trajectory Judge Agent separately audits tasks with 100% rollout failure rates, classifying them as TOO_HARD (retained) or DESIGN_FLAW (discarded); 94 tasks (~2%) were removed this way.

SETA-Evol takes the SETA-Synth pool and applies model-calibrated evolution operators per environment based on the training model's pass rate \(r\):

Source Difficulty \(r\) range Strategy
Easy \(r > 0.5\) Difficulty Increase
Medium \(r \in (0, 0.5]\) Context Shift
Difficult \(r = 0\) (partial) Difficulty Decrease

Evolved tasks pass the same Docker/no-op/oracle/self-review pipeline and are re-evaluated on the training model to confirm difficulty shift.

RL Training uses GRPO with asymmetric clipping and a partial-progress reward: $\(r = \frac{\text{\# tests passed}}{\text{\# total tests}} + 0.2 \cdot \mathbf{1}\!\left[\frac{\text{\# tests passed}}{\text{\# total tests}} = 1\right]\)$ This raises the fraction of tasks providing meaningful reward variance from 49% to 86%. Dynamic sampling (DAPO-style) filters uniform-reward batches.

Key Contributions

  • SETA-Synth: source-adaptive synthesis pipeline decoupling per-source format adaptation (adapter prompt) from general task construction (base prompt + Datapoint Agent), enabling extension to new sources without modifying the core pipeline.
  • SETA-Evol: difficulty-adaptive evolution framework that selects per-environment mutation operators calibrated to the training model's live pass rate, shifting data toward the productive RL zone (neither trivially solvable nor totally infeasible).
  • SETA-Env: 4,567 verified terminal environments (3,255 synthesized + 1,312 evolved) spanning 14 tech-stack categories, the largest open-source verifiable terminal RL dataset; generated using the Claude Agent SDK with Claude Opus 4.6.
  • SOTA RL results at 8B scale: Qwen3-8B achieves 12% pass@1 on Terminal-Bench 2.0 (3.3× over base, best reported for RL-trained 8B models); cross-family gains on DeepSeek-V4-Flash.

Results

  • Qwen3-8B SETA (RL): 10.7±1.3% mean / 12% best-run pass@1 on Terminal-Bench 2.0 vs. 3.1±0.6% base; 17.8±1.2% on Terminal-Bench 1.0 vs. 7.8±1.2% base.
  • Comparison at 8B RL scale: outperforms Endless Terminals-8B (6.7%) and OpenThoughts-Agent-v1-RL (4.9%); approaches Nemotron-Terminal-8B SFT (13.0%) trained on 254k+ trajectories.
  • DeepSeek-V4-Flash (CAMEL Terminal Agent harness): pass@1 40% → 43.0±2.5%; pass@5 54% → 58% on Terminal-Bench 2.0.
  • Generalization (Qwen3-8B, pass@4): CompileBench 6.7% → 40%; CRUST-Bench 15% → 24%.
  • SETA-Evol validity: median Qwen3-8B pass-rate shifts from 6% → 38% for SLIGHT_DECREASE (n=505) and 83% → 69% for INCREASE_DIFFICULTY (n=143); 77% and 60% of pairs respectively move in the declared direction.

Limitations

  • Only 560 of 4,567 environments are used for training runs due to compute constraints; filtering for base-model difficulty may exclude signal useful for stronger models.
  • Thinking mode disabled for Qwen3-8B training because extended chain-of-thought frequently exceeded the maximum response length, invalidating training trajectories.
  • Category distribution is imbalanced (system-administration 35.8%, audio-processing 0.0%), reflecting source data bias that SETA-Evol only partially mitigates.
  • Post-rollout Trajectory Judge itself is an LLM and may misclassify design flaws vs. genuinely hard tasks (only a 2% audit rate was reported, so the false-negative rate is unknown).
  • The Trajectory Judge operates at 100% failure rate; tasks with low but nonzero pass rates that are nonetheless broken may slip through.

Relevance to Harnesses / Meta-Harnesses

SETA is itself a multi-agent meta-harness: it orchestrates an Idea Agent, Datapoint Agent, and Trajectory Judge Agent in a structured pipeline to manufacture and validate RL training environments at scale. The evaluation loop depends directly on the CAMEL Terminal Agent harness (ChatAgent + Terminal Toolkit), which is used both to generate rollout trajectories during training and as the evaluation framework for Terminal-Bench 2.0; harness configuration (context window, iteration cap) is a first-class experimental variable. The Harbor framework provides the unified task-format substrate that makes the synthesis and evolution pipelines interoperable. For researchers tracking harnesses and meta-harnesses, SETA exemplifies a pattern where the harness is not just an evaluation wrapper but the generative backbone through which training environments are produced, validated, and difficulty-calibrated.