Skip to content

The Remarkable Effectiveness of Providing AI Agents with Natural Language Tools: A Replication Study Validating NLT Performance Across 14 Models

🕒 Published (v1): 2026-07-04 16:59 UTC · Source: Arxiv · link

Why this paper was selected

Replicated NLT across 14 models; directly answers structured-vs-NL tool calling for harness builders

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper independently replicates Johnson et al. (2025)'s Natural Language Tools (NLT) framework, which replaces JSON-schema-based structured tool calling with free-form YES/NO natural language tool selection. Across 14 models and 8,560 trials, NLT yields +14.9 pp accuracy gain and a 93% reduction in critical errors versus structured calling, confirming the original findings while revealing that the advantage is capability-dependent: smaller and reasoning models gain most; heavily RL-optimized frontier models (GPT-5, Gemini 2.5 Pro) show near-parity or reversal.

Problem

Agentic systems canonically use structured (JSON) tool calling, which the original Johnson et al. (2025) work argued imposes a cognitive/distributional bottleneck—forcing models to draw on coding-corpus JSON-generation priors while simultaneously solving domain tasks, fragmenting representational resources. The original study's claim (+18.4 pp over 10 models) required independent validation with a broader, newer model set (frontier, reasoning, open-weight) and a separate open-source harness to rule out implementation artifacts.

Method

The authors built an independent Python evaluation harness (evaluator.py, parser.py, client.py, scenarios.py) using only urllib—no external API libraries—that implements the same \(2 \times 2 \times 2\) factorial design as the original: Approach (NLT vs. Structured) × Scenario (Alex customer service / Sage mental health) × Perturbation (non-perturbed / semantically-equivalent rephrasing). NLT expresses tools as a plain-language YES/NO checklist; structured uses OpenAI-compatible function-calling JSON schemas. Each model × condition cell runs 5 independent replicates over 16 inputs → 640 trials per model. Accuracy is exact-match; a survivorship-bias correction zeros out any condition where \(\geq 70\) of 80 trials produce errors (affects 8 of 107 entries: all Mistral-7B and Qwen3-VL structured conditions).

Key Contributions

  • First independent, open-source replication of NLT, implemented from scratch with different tooling than Johnson et al. (2025).
  • Extended evaluation to 14 models (+5 not in original): GPT-5, Claude Sonnet 4, Gemini 2.5 Pro, DeepSeek-R1, Kimi-K2.
  • Per-model capability-moderation analysis revealing a systematic gradient: models without native tool calling or with chain-of-thought reasoning gain most; RL-optimized frontier models gain least or reverse.
  • Survivorship-bias correction methodology for error-heavy structured conditions.
  • Open-source harness, raw results, and all prompts released for reproducibility.

Results

  • Overall accuracy: NLT 62.3% vs. Structured 47.4% (corrected); \(\Delta = +14.9\) pp (vs. +18.4 pp in original).
  • Error reduction: NLT 51 errors vs. Structured 755 errors across all trials — 93% fewer errors.
  • Token usage: NLT 3.38M tokens vs. Structured 4.52M tokens — 25.2% reduction (vs. 31.4% in original).
  • NLT wins in 11 of 14 models.
  • Largest NLT gains: Claude Sonnet 4 +43.1 pp (61.9% vs. 18.8%), Mistral-7B +39.4 pp (39.4% vs. 0%), Qwen3-VL +33.8 pp (corrected), DeepSeek-R1 +24.0 pp.
  • Near-parity or reversal: GPT-5 +1.6 pp, Kimi-K2 −0.6 pp, GPT-OSS-120B −6.4 pp, Gemini 2.5 Pro −33.7 pp (strongest structured outlier at 82.1% structured vs. 48.3% NLT).
  • Domain: Sage (mental health) shows larger gain (+16.1 pp) than Alex (customer service, +13.4 pp).
  • Perturbation robustness: NLT stable at 62.4% (non-perturbed) vs. 62.2% (perturbed); structured 46.1% vs. 48.8%.
  • Variance: NLT (0.1913) slightly higher than structured (0.1702); original claim of 70% variance reduction not replicated (confounded by systematic structured failures compressing structured variance).

Limitations

  • Only single-turn, parameterless tool selection is tested; multi-turn interactions and parameterized calls are not replicated.
  • Two models have partial data: Gemini 2.5 Pro (6/8 conditions) and Qwen3-VL (5/8 conditions) due to API availability gaps.
  • API latency variability (up to 120 ms across providers) may affect results; infrastructure differences from the original study are uncontrolled.
  • Evaluated scenarios are limited to two domains (customer service, mental health) with 16 inputs each; generalization beyond these is untested.
  • Temperature fixed at 1.0; no ablation over sampling parameters.
  • Variance reduction claim from the original study could not be fairly evaluated due to survivorship bias in structured failures compressing measured variance.

Relevance to Harnesses / Meta-Harnesses

This paper directly evaluates an alternative tool-dispatch interface for agentic harnesses: replacing JSON function-calling schemas with natural-language YES/NO tool selection. The finding that NLT cuts errors by 93% and token overhead by 25% is immediately actionable for harness architects deciding how to route tool calls across sub-agents—particularly relevant in recursive or multi-agent pipelines where a single structured-call failure cascades into retries, fallback routing, and coordination overhead. The capability-moderation result (small/reasoning models gain most; RL-optimized models may prefer structured) provides a practical decision rule for heterogeneous model fleets managed by a meta-harness, suggesting that the dispatcher should select the tool-calling interface dynamically based on model type rather than applying one strategy uniformly.