Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading¶

🕒 Published (v1): 2026-07-09 21:56 UTC · Source: Arxiv · link

Why this paper was selected

Long-horizon terminal benchmark with dense reward grading; raises bar beyond final-outcome-only agent evals

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Long-Horizon-Terminal-Bench (LHTB) is a 46-task terminal benchmark designed to stress-test LLM agents on workflows requiring hundreds of steps and up to 90 minutes of execution. Unlike prior binary-graded benchmarks, LHTB decomposes each task into fine-grained subtasks with a weighted partial-credit reward, enabling dense intermediate feedback. Even the strongest tested model (GPT-5.5) solves only 15.2% of tasks at a 0.95 reward threshold, revealing a large capability gap for long-horizon agentic execution.

Problem¶

Existing terminal and software-engineering benchmarks (e.g., SWE-Bench, Terminal-Bench) evaluate short, narrowly scoped tasks under binary pass/fail grading. This design yields two compounding problems: (1) tasks finish in minutes and understate the difficulty of multi-stage real workflows requiring hours of iterative planning and debugging; (2) outcome-only grading produces extremely sparse reward signals, rendering an agent that completes 90% of steps indistinguishable from one that immediately fails.

Method¶

Each LHTB task follows the Harbor/Terminal-Bench-2 containerized format—a natural-language instruction, a Docker image with all assets, a task config, and an oracle—but is further decomposed into $K$ semantically meaningful subtasks $\{s_1, \ldots, s_K\}$. The task reward is the weighted average of normalized per-subtask scores:

\[R = \frac{\sum_{k=1}^{K} w_k r_k}{\sum_{k=1}^{K} w_k}, \quad r_k \in [0, 1]\]

Three subtask types are used: binary (Boolean programmatic checks), continuous/thresholded (e.g., linear decay from 1.0 as metric error grows), and episode-aggregating (fraction of game levels or audit episodes passed). Reward weight is concentrated on hidden stress cases—dynamically generated harder inputs (renamed fields, gzip+base64 wrappers, injected noise, rotated images)—so agents cannot score highly by hard-coding the public test cases. Tasks were difficulty-calibrated by iterative design against DeepSeek-V4-Pro under 1.5-hour budgets; 120 candidate tasks were filtered to 46 final tasks. Evaluation uses the Terminus-2 agent harness (Codex for GPT-5.3) across 15 frontier models.

Key Contributions¶

46 long-horizon terminal tasks spanning 9 categories (software/reverse engineering, earth/climate/energy, multimodal/imaging, scientific computing, research reproduction/ML, systems/performance/security, APEX professional workflows, interactive games, logic/constraint puzzles).
Dense subtask-based grading with partial credit that separates near-misses from complete failures, resolving ties invisible to binary evaluation.
First large-scale evaluation of 15 frontier models on tasks averaging 231 episodes, 9.9M tokens, and 85.3 minutes per run.
Cost–reward Pareto analysis showing that per-task inference cost ($2.5–$28) does not reliably predict pass rate.
Quantitative failure-mode decomposition: timeouts vs. early exits (overconfidence) vs. harness errors.

Results¶

GPT-5.5 achieves the best pass@1: 15.2% at $R \geq 0.95$ (7/46 tasks) and 10.9% at $R = 1.0$ (5/46).
Mean pass@1 across all 15 models: 4.3% at $R \geq 0.95$; 1.7% at $R = 1.0$.
Under strict binary grading ($R = 1.0$), 10 of 15 models pass zero tasks.
Second tier: MiniMax M3, Kimi K2.7 Code, DeepSeek V4 Pro each at 6.5% ($R \geq 0.95$); Kimi K2.6 and Grok 4.20 solve zero tasks at this threshold.
32.9% of all 690 model–task runs make no meaningful progress ($R < 0.05$); 62.8% achieve partial reward that binary grading would collapse to zero.
Near-misses ($0.75 \leq R < 0.95$) occur more than twice as often as passes (73 vs. 30 runs).
79% of unresolved runs are timeouts (agent still working at 90-min cutoff), mean reward 0.10–0.35; early exits account for 19% and reflect overconfidence.
Average cost: $10.2/task; range $2.47 (Hy3) to $27.57 (GPT-5.4); pass rate and cost are not monotonically related.
Spearman $\rho = 0.56$ between pass rate and mean reward, indicating they capture complementary aspects of agent capability.

Limitations¶

Only 46 tasks; coverage across the 9 categories is uneven (3–7 tasks each), limiting statistical power for category-level conclusions.
A single 90-minute timeout per run; some tasks may be solvable with more time, conflating budget limits with true capability ceilings.
Pass@1 only (single rollout per model–task pair); variance across runs is not measured.
Paper is truncated; full failure-mode analysis and ablations may contain additional findings not summarized here.
Benchmark size and evaluation cost (~$10/task × 15 models × 46 tasks ≈ $7,000) make iterative development on LHTB expensive.
Hidden stress suite prevents public reproducibility of exact verifier logic, which limits independent auditing of scores.

Relevance to Agentic AI / LLM Agents¶

LHTB directly probes the frontier bottleneck for LLM agents: sustaining coherent, multi-hour action sequences in open-ended terminal environments rather than solving isolated, short-horizon coding problems. The dense reward design addresses a core methodological gap in agentic evaluation—sparse outcome signals are insufficient to guide research when most agents fail—and the benchmark's separation of timeouts from early exits operationalizes a new failure category (overconfident self-termination) that is otherwise invisible. The cost–reward Pareto analysis is practically important for researchers choosing frontier models as agent backbones. By showing that even the strongest model achieves only 15.2% success, LHTB establishes a concrete ceiling that motivates work on long-context state management, iterative self-verification, and multi-stage planning in agentic systems.