Skip to content

ResiliBench: Evaluating Agentic Workflow Adaptation in Stochastic Environments

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR ResiliBench; evaluates workflow adaptation under instruction variability and tool uncertainty

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ResiliBench is a benchmark of 5,040 tasks that stress-tests LLM workflow agents under two realistic adversarial conditions: probabilistic tool failures (modeled via parameterized error simulators) and deliberately degraded workflow instructions (seven categories of systematic perturbations). It provides MDP-derived optimal workflow prompts as a theoretical ceiling and evaluates how far current models fall from that ceiling.

Problem

Existing tool-use benchmarks treat API unreliability and instruction ambiguity as noise to be filtered out rather than phenomena to be measured. There is no systematic framework for evaluating how well LLM agents adapt—recover from API errors, replan when given flawed instructions—under controlled, reproducible stochastic conditions. The gap between an ideal workflow plan and real-world execution resilience is unmeasured.

Method

ResiliBench constructs tasks via a three-stage automated pipeline:

  1. Tool library generation: 30 APIs across 6 categories (data processing, file operations, network, computation, integration, utility), each with explicit probabilistic error models covering five failure modes (INVALID_INPUT, OPERATION_FAILED, TIMEOUT, CALCULATION_ERROR, OVERFLOW) and a base success rate \(\rho_{\text{base}} = 0.8\).

  2. Task generation: RAG-based semantic matching maps operation sequences (e.g., [read, validate, transform, aggregate, write]) to concrete tools, yielding 5,040 tasks across 5 types and 3 complexity levels.

  3. Prompt generation: Four prompt variants per task—(a) baseline, (b) Chain-of-Thought, (c) MDP-optimal workflow, (d) flawed workflow. The MDP-optimal prompt is derived by training a PPO policy with Transformer networks over the MDP state space (tracking tool execution states and progress), using a two-phase reward: coverage-first then sequence-optimized. Flawed prompts inject perturbations across 7 categories: sequential ordering errors, tool misuse, parameter errors, missing steps, redundant operations, logic discontinuity, and semantic drift.

Evaluation scores tasks on four criteria simultaneously: required tool coverage (must be 100% for full success), sequence correctness, output generation, and explicit completion signals.

Key Contributions

  • A benchmark explicitly centering stochastic tool behavior and instruction-quality degradation as first-class evaluation axes, not incidental artifacts.
  • An MDP formulation (solved via PPO) that computes theoretically optimal workflows maximizing expected success rate given known tool reliability statistics—serving as an analytically grounded upper bound.
  • Seven-category systematic perturbation framework for flawed workflow prompts, enabling fine-grained attribution of model fragility.
  • Real-world extension integrating 23 live public APIs empirically validated for non-deterministic behavior (each invoked 20 times to confirm stochasticity), with 8 multi-step sequential tasks.
  • Scaling analysis revealing emergent, non-linear workflow competence in the Qwen2.5 series.

Results

  • MDP-optimal prompts: average full-success rate 62.1%; CoT prompts: 50.8%; flawed prompts: 54.3% (averaged across all models).
  • Best model on optimal prompt: GPT-4o-mini (67.7%), followed by Llama-3.3-70B (66.1%), Qwen2.5-32B (65.0%).
  • Instruction-quality robustness is highly model-dependent: GPT-4o-mini drops only 5.5 pp (optimal 67.7% → flawed 62.2%); Gemini-2.5-Flash drops 40.1 pp (60.1% → 20.0%).
  • Task complexity degrades performance consistently: GPT-4o-mini drops from 72.4% (simple) to 53.7% (hard computation pipelines).
  • Emergent scaling in Qwen2.5: 0.5% at 3B → 63.5% at 7B → 65.0% at 32B/72B (non-linear jump).
  • Real-world API experiments produce patterns consistent with simulated results.

Limitations

  • The simulated tool library (30 APIs, fixed 6 categories) is narrow; generalization to arbitrary real-world API ecosystems is unverified.
  • The MDP upper bound assumes known tool failure probabilities; agents evaluated without this knowledge, so the bound is not tight in practice.
  • Evaluation caps at 10 conversational turns, which may penalize deliberate but slow recovery strategies.
  • Flawed prompt perturbations are synthetically injected; they may not fully capture the distribution of real user instruction failures.
  • Only 8 real-world tasks across 23 APIs—too small to draw statistically robust conclusions.

Relevance to Harnesses / Meta-Harnesses

ResiliBench directly benchmarks the resilience layer that any production agentic harness must implement: what happens when an underlying tool call fails mid-workflow and the orchestrator must replan. The MDP-optimal workflow generation framework is itself a meta-harness primitive—a planner that reasons over tool reliability statistics to emit maximally robust execution plans. The seven-category perturbation taxonomy maps directly onto failure modes a harness designer must handle (missing steps, parameter misconfiguration, semantic drift in sub-agent instructions). The benchmark's finding that CoT prompts underperform even flawed workflow prompts on average (50.8% vs. 54.3%) suggests that explicit structured planning—exactly what harnesses provide—is more valuable than reasoning-in-context alone.