Skip to content

Small Language Models for Efficient Agentic Tool Calling: Outperforming Large Models with Targeted Fine-tuning

🕒 Published (v1): 2025-12-17 20:12 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

A fine-tuned OPT-350M (350M parameter) model achieves 77.55% pass rate on ToolBench, outperforming ChatGPT-CoT (26%), ToolLLaMA-DFS (30%), and Claude-CoT (2.73%) by large margins. The work demonstrates that task-specific SFT on a domain-matched dataset can substitute for scale in structured tool-calling tasks. The primary implication is dramatic cost reduction for enterprise agentic deployments.

Problem

LLM-driven agentic tool-calling workflows are prohibitively expensive at production scale due to inference costs, API dependencies, and data privacy risks. Prior work on small models either evaluated narrow domains or lacked systematic comparison across diverse tool-manipulation categories. The trade-off between model size, training efficiency, and tool-calling performance was not well characterized.

Method

Fine-tune facebook/opt-350m (350M params) via Supervised Fine-Tuning (SFT) using Hugging Face TRL's SFTTrainer on the ToolBench dataset (187,542 training examples derived from 16,000+ RapidAPI Hub APIs). Training used a single epoch on Amazon SageMaker (ml.g5.8xlarge) with: lr=5×10⁻⁵, 100 warmup steps, effective batch size 32 (4-step gradient accumulation), gradient clipping (max norm=0.3), FP16 mixed precision, gradient checkpointing, AdamW with weight decay 0.01. The model learns Thought-Action-Action Input structured output patterns. Evaluation uses ToolEval (ChatGPT-as-judge, ≥4 assessments per query, majority voting) across 1,100 queries in 6 ToolBench categories.

Key Contributions

  • Demonstrates that a 350M parameter SLM, fine-tuned for one epoch on ToolBench, outperforms models 20–500× larger (including 175B ChatGPT) on structured tool-calling benchmarks.
  • Provides per-category breakdown across all six ToolBench splits (G1-instruction/category/tool, G2-category/instruction, G3-instruction), showing consistent 74–80.5% pass rates.
  • Argues empirically that "parameter dilution" in large general-purpose models is a liability for structured, format-constrained tasks like tool calling.
  • Presents a cost-effective training recipe (single epoch, commodity cloud GPU, open-source toolchain) reproducible by resource-constrained organizations.

Results

  • Overall pass rate: OPT-350M fine-tuned → 77.55% vs. ToolLLaMA-DFS (7B) → 30.18%, ChatGPT-CoT (175B) → 26.00%, ToolLLaMA-CoT (7B) → 16.27%, Claude-CoT (52B) → 2.73%.
  • Per-category (Ours / ToolLLaMA-DFS / ChatGPT-CoT):
  • G1-instruction: 78.5 / 32.5 / 33.0
  • G1-category: 74.0 / 32.5 / 29.5
  • G1-tool: 79.0 / 28.0 / 29.5
  • G2-category: 80.5 / 32.5 / 24.5
  • G2-instruction: 74.5 / 29.5 / 24.0
  • G3-instruction: 80.0 / 22.0 / 5.0
  • Performance variance across categories is only 6.5 percentage points (74.0–80.5%), indicating consistent generalization within ToolBench.

Limitations

  • Benchmark overfitting: Model is optimized specifically for ToolBench's evaluation criteria and format; generalization to other tool-calling frameworks or real-world API ecosystems is unverified.
  • Limited contextual reasoning: 350M parameter capacity may be insufficient for tool selection embedded within complex, ambiguous conversational contexts requiring multi-step pre-reasoning.
  • Controlled API scope: Evaluation covers a fixed API set; real-world deployments involve dynamic, interdependent, authenticated tool ecosystems not represented in ToolBench.
  • Training data dependency: Performance is bounded by ToolBench data quality and coverage; novel API designs outside the training distribution may cause brittle behavior.
  • Maintenance burden: Specialized fine-tuning requires retraining as APIs evolve, unlike larger models that may adapt via few-shot prompting.
  • ToolEval validity: The ChatGPT-as-judge evaluation framework introduces its own biases; reported numbers depend on this automated evaluator's reliability.

Relevance to Agentic AI / LLM Agents

This paper directly challenges the default assumption in agentic AI that capable tool-using agents require large, general-purpose models, demonstrating instead that SFT on task-specific trajectory data can yield a 350M model that outperforms GPT-scale baselines on structured API orchestration. For researchers tracking LLM agents, the result is relevant to the open question of how to deploy cost-efficient agents in enterprise settings without sacrificing tool-calling fidelity. The Thought-Action-Action Input training format aligns with the ReAct paradigm central to modern agent frameworks, making the approach directly composable with existing agent scaffolds. The findings also raise important questions about benchmark validity—whether ToolBench pass rate under ChatGPT-as-judge truly reflects real-world agentic capability or rewards format compliance.