ResiliBench: Evaluating Agentic Workflow Adaptation in Stochastic Environments

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Benchmarks agent workflow adaptation under stochastic tool failures; novel resilience test

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ResiliBench is a benchmark for evaluating LLM workflow execution under two orthogonal stressors: probabilistic tool failures (API errors) and instruction quality variability (flawed workflow prompts). It provides 5,040 tasks, a 30-API tool registry with explicit error models, and MDP-derived optimal workflows as a theoretical ceiling. Evaluation across proprietary and open-source models reveals dramatic robustness disparities and emergent multi-step tool-use abilities at model-size thresholds.

Problem

Existing tool-use benchmarks treat API unreliability and instruction ambiguity as noise to be filtered out, not as primary evaluation dimensions. As a result, they do not measure how well LLM agents adapt when tool calls fail stochastically or when the workflow plan they receive is subtly wrong — both pervasive conditions in real production deployments.

Method

ResiliBench has three core components:

1. Tool Registry: 30 canonical APIs organized into 6 categories (data processing, file operations, network, computation, integration, utility), each with probabilistic error models covering five failure types (INVALID_INPUT, OPERATION_FAILED, TIMEOUT, CALCULATION_ERROR, OVERFLOW) and a base success rate \(\rho_{\text{base}} = 0.8\).

2. Task Generation: 5,040 tasks auto-generated via RAG-based semantic matching across five task types and three complexity levels; optionally enhanced by an LLM for description quality.

3. Prompt Variants: Each task ships with four prompt types:

4. Baseline: task description + tool instructions
5. Chain-of-Thought (CoT): adds explicit step-by-step reasoning instructions
6. MDP-Optimal: adds an execution plan derived from a PPO-trained policy on an MDP that maximizes expected cumulative reward given known tool failure probabilities; state captures execution history and progress; action space is structured tool invocations; training uses curriculum learning across five difficulty stages
7. Flawed Workflow: MDP-optimal plan with one of seven controlled perturbations (sequential ordering errors, tool misuse, parameter errors, missing steps, redundant operations, logic discontinuity, semantic drift)

Evaluation uses a multi-turn interactive environment (up to 10 turns) with one-tool-per-turn execution, grading tasks on four criteria: Required Tools Coverage, Sequence Correctness, Output Generation, and Explicit Completion Signals.

Key Contributions

• ResiliBench benchmark with 5,040 tasks systematically covering instruction quality and tool uncertainty as first-class evaluation dimensions
• MDP-based workflow generation framework (PPO + Transformer, curriculum learning) that produces theoretically grounded optimal execution plans as evaluation anchors
• Seven-category controlled perturbation taxonomy for flawed workflow generation
• Real-world extension integrating live public APIs (23 APIs, 8 tasks) empirically selected for stochastic behavior
• Evidence of emergent multi-step tool-use ability: Qwen2.5 series jumps from 0.5% (3B) to 63.5% (7B) then plateaus

Results

• MDP-optimal prompts: average full-success rate 62.1%; best: GPT-4o-mini 67.7%, Llama-3.3-70B 66.1%, Qwen2.5-32B 65.0%
• CoT prompts: average 50.8% full success
• Flawed workflow prompts: average 54.3% full success (below CoT despite containing more information, due to error injection)
• Robustness disparity: GPT-4o-mini drops only 5.5 pp (optimal 67.7% → flawed 62.2%); Gemini-2.5-Flash drops 40.1 pp (optimal 60.1% → flawed 20.0%)
• Complexity degradation: GPT-4o-mini degrades from 72.4% (simple) to 53.7% (complex computation pipelines); similar patterns across all tested models
• Scaling emergence (Qwen2.5): 3B → 0.5%, 7B → 63.5%, 32B → 65.0%, 72B → 65.0% — non-linear jump between 3B and 7B
• Semantic drift is the most damaging perturbation type; advanced models are most resilient to ordering and parameter errors

Limitations

• MDP formulation models tool failures as stationary and independent; real APIs exhibit correlated failures and non-stationary reliability
• 30-API tool library is synthetic; diversity and coverage may not match production API ecosystems
• Evaluation is limited to 10 conversational turns, constraining measurable recovery depth
• Real-world extension is small (8 tasks, 23 APIs) and cannot replace the scale of the simulated suite
• Binary/ternary success grading (full/partial/fail) may not capture nuanced partial recovery trajectories
• Models evaluated at a single temperature/configuration; hyperparameter sensitivity is not reported

Relevance to Agentic AI / LLM Agents

ResiliBench directly addresses the gap between benchmark performance and real-world agentic deployment by making stochasticity and instruction imperfection first-class evaluation axes — critical for building reliable tool-using agents. The MDP-optimal workflow as a principled ceiling (upper bound iii: policy knows tool procedures and failure probabilities but not random seeds) is a meaningful contribution to agentic evaluation methodology, moving beyond "can the model call the right tool" to "can the model adaptively replan under uncertainty." The finding that robustness to flawed instructions is a distinct capability dimension orthogonal to raw performance has direct implications for RLHF/instruction-tuning priorities for agentic systems. The emergent multi-step tool-use result reinforces the scaling-law literature for agent-specific capabilities and sets a data point for minimum useful model size in workflow automation contexts.