Towards Reliable Benchmarking: A Contamination Free, Controllable Evaluation Framework for Multi-step LLM Function Calling

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Tom Mitchell (CMU); contamination-free, controllable benchmark for multi-step function calling

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FuncBenchGen is a synthetic benchmark generation framework that casts multi-step LLM tool use as traversal over a hidden function-dependency DAG, enabling contamination-free and fine-grained controllable evaluation of tool-augmented language models (TaLMs). It exposes that performance degrades sharply with dependency depth and that type-compatible distractor functions are the most damaging irrelevant signals. A lightweight mitigation — restating prior variable values at each step — yields substantial gains (e.g., GPT-5: 62.5% → 81.3% success rate).

Problem

Existing function-calling benchmarks (API-Bank, BFCLv4, ToolBench, ComplexFuncBench, LongFuncEval) suffer from two compounding problems: (1) data contamination — task Q&A pairs are publicly accessible and may overlap with pretraining corpora or be retrieved via test-time web search; (2) limited difficulty controllability — none of the prior benchmarks control all three of: required function set size, dependency depth, and irrelevant-function type-compatibility. This prevents systematic understanding of which factors drive model failures.

Method

FuncBenchGen generates synthetic, contamination-free evaluation tasks at evaluation time via two stages:

1. Graph Structure Generation: A directed acyclic graph (DAG) \(G = (F, E)\) is constructed with \(n_\text{core}\) core nodes at target dependency depth \(d\), plus \(n_\text{conn}\) connected irrelevant nodes (CINs — share type-compatible variables with core nodes) and \(n_\text{dis}\) disconnected irrelevant nodes (DINs — isolated or inter-connected but not touching the core DAG).

2. Function Schema Creation: Each DAG node is converted into a function schema with a randomly generated name, typed input/output variables, and a natural-language description. Functions use three-digit integer values; they return correct output only given exact expected inputs (incorrect inputs yield random wrong outputs, simulating silent API failures).

The agent receives only the function list and known input variable values — not the DAG itself — and must infer the call sequence to compute the target variable \(v_T\). The interaction is multi-turn: at each step the agent may invoke multiple functions, receives their outputs, and continues until it makes no further calls.

Key Contributions

• FuncBenchGen framework: automated generation of contamination-free multi-step tool-use tasks with independently controllable parameters: \(n_\text{core}\), \(d\), \(n_\text{conn}\), \(n_\text{dis}\).
• Systematic evaluation of 7 SoTA models (GPT-5, GPT-5-mini, Gemini-2.5-Pro, Gemini-2.5-Flash, Qwen3-235B, GPT-4.1, GPT-4.1-mini) across the full complexity grid.
• Key findings: reasoning-optimized models consistently outperform general-purpose models; CINs (type-compatible distractors) are far more harmful than DINs; shallower/more sequential DAGs are easier; models produce syntactically valid calls but fail to propagate argument values across steps.
• State-restatement mitigation: at each step, prior variable values are explicitly restated to the agent; improves GPT-5 success rate from 62.5% to 81.3%.

Results

• GPT-5 achieves 72.5% / 38.2% / 15.0% success at 5 / 10 / 20 core nodes; all other models fare substantially worse.
• GPT-4.1 scores 12.0% at 5 core nodes; GPT-4.1-mini scores 0.0% at 20 core nodes.
• Gemini-2.5-Pro: 46.5% → 6.0% as core nodes scale from 5 to 20.
• At dependency depth 1 (star graph, 10 core nodes), GPT-5 approaches 90% success; drops below 30% for depths 4–8.
• Connected irrelevant nodes uniformly degrade all models most severely; disconnected nodes have mixed effects (GPT-5 slightly improves with DINs; Gemini-2.5-Pro worsens).
• GPT-5 at 40 core nodes (disconnected setting) falls below 10%, showing even the best model is not ready for large real-world API sets (e.g., MCP servers).
• Sufficient thinking budget is critical: minimal thinking budget drops GPT-5 below 50% except in the zero-distractor condition.
• State-restatement mitigation: GPT-5 62.5% → 81.3%; substantial gains observed across all evaluated models.

Limitations

• Evaluation uses synthetic, integer-valued functions with deterministic logic — does not capture real-world API complexity (auth, pagination, partial failures, natural-language ambiguity).
• Function connectivity is determined by type/subtype matching alone, a simplified proxy for semantic reasoning in real APIs.
• Maximum call count is capped at \(2\times\) the minimum required, which may undercount failures for models that are slower to give up.
• Only seven models evaluated; results on smaller open-source models or fine-tuned tool-use specialists are absent.
• The state-restatement mitigation is evaluated only on success rate; latency, token cost, and scalability to very long sequences are not analyzed.

Relevance to Agentic AI / LLM Agents

This work directly addresses a core reliability problem in agentic evaluation: without contamination-free, controllable benchmarks, it is impossible to know whether apparent capability gains reflect genuine tool-use reasoning or benchmark leakage. The DAG-based formulation captures the essential structure of real agentic workflows (sequential API chaining, dependency inference from signatures only) while enabling the kind of ablation studies needed to understand failure modes. The finding that state tracking — not function selection — is the primary bottleneck in multi-step tool use shifts the research agenda toward memory/context management in agent loops. The demonstrated effectiveness of explicit variable restatement is a directly actionable design principle for agentic system prompts and scaffolding.