Skip to content

Diagnosing and Calibrating Tool-Call Boundary Drift in Multi-Teacher On-Policy Distillation

🕒 Published (v1): 2026-07-15 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

Diagnoses tool-call boundary drift in multi-teacher distillation; novel problem for agentic LM training

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Multi-teacher on-policy distillation (MOPD) for tool-using LLMs silently shifts the student's tool-call boundary: vanilla GKD improves should-call recall but also increases spurious tool invocations on should-respond examples, a drift invisible to aggregate loss metrics. The authors diagnose this as a behavior-leverage imbalance at mode-entry tokens and introduce Soft Clamp, a localized token-level divergence compressor that reduces over-calling from 14.2±2.1% to 9.0±0.2% while preserving most of the call-recall gain over SFT.

Problem

When a student learns from two frozen specialized teachers—one for tool calls, one for direct responses—via generalized knowledge distillation (GKD) on its own rollouts, the tool/no-tool decision boundary drifts toward over-calling. Crucially, standard aggregate diagnostics (per-token JSD, token exposure counts, teacher entropy, top-\(k\) overlap) do not flag the drift: in representative runs, response examples receive more token exposure and larger full-sequence JSD than tool-call examples, yet the student's tool-entry probability on should-respond examples still increases. This leaves practitioners without a principled diagnostic or mitigation.

Method

The paper frames the problem as behavior-boundary calibration through the lens of behavior leverage: mode-entry tokens such as <tool_call> and function names constrain all subsequent trajectory tokens, so even a small local learning signal at these positions can redirect generation mode. Four calibration strategies are compared:

  1. Hard Clip – truncates token divergence above a fixed threshold, zeroing the gradient beyond it.
  2. Global Reweight – rescales all token losses by batch-relative divergence (adapted from GNDPO).
  3. Soft Clamp (proposed) – computes a batch-adaptive threshold \(C = \text{stopgrad}(k \cdot \text{mean}_i(d_i))\) and applies: $\(d'_i = \begin{cases} d_i, & d_i \leq C \\ d_i \cdot \text{stopgrad}\!\left(\tfrac{C}{d_i}\right), & d_i > C \end{cases}\)$ For extreme tokens the forward value is capped at \(C\) while the gradient is scaled by \(C/d_i\)—nonzero, unlike hard clipping. Ordinary tokens are left unchanged.
  4. Inference-time entry bias – validation-tuned logit bias on <tool_call> at decode time.

Diagnostics measure JSD concentration (fraction of total JSD in the top 1%/5%/10% of tokens by divergence) and decision-pressure metrics (P(<tool_call>) on should-respond examples), evaluated across three seeds.

Key Contributions

  • Failure discovery: Reproducible tool-call boundary drift in vanilla MOPD; should-respond recall degrades despite benign aggregate statistics.
  • Diagnostic framework: Links sparse high-divergence token concentration at mode-entry positions—rather than aggregate exposure or total JSD—to the observed behavior shift; validates with targeted-clamp ablations showing first-four-token clamping alone is insufficient.
  • Calibration comparison: Exposes trade-offs among Hard Clip, Global Reweight, Soft Clamp, and inference-time entry bias across gradient preservation, locality, seed variability, call recall, and deployment cost.
  • Soft Clamp: Batch-adaptive, localized divergence compressor; reduces top-1% JSD share from 41.2±0.4% to 15.4±3.2% and exhibits the lowest over-calling standard deviation among GKD variants.

Results

All on Qwen3.5-9B, APIGen-MT-derived decision set, mean±std over 3 seeds for GKD rows:

  • Over-calling: Vanilla GKD 14.2±2.1% → Hard Clip 11.9±0.9% → Global Reweight 9.7±0.8% → Soft Clamp 9.0±0.2% (vs. Base SFT 4.9%)
  • Decision accuracy: 88.6±0.3% (vanilla) vs. 88.7±0.7% (Soft Clamp); comparable
  • Call recall: Vanilla GKD 91.5±1.7% → Soft Clamp 86.5±1.4% (above Base SFT 75.5%)
  • Respond recall: 85.8±2.1% (vanilla) → 91.0±0.2% (Soft Clamp)
  • Soft Clamp beats vanilla GKD in over-calling in all 3 paired seeds (by 3.6, 4.7, 7.3 pp)
  • BFCL multi-turn diagnostic (800 tasks, 3136 turns): Soft Clamp gives lowest calls/turn (1.289±0.023), loop@5 (5.0±0.3%), and repeat-call rate (11.3±0.6%) among GKD variants
  • Inference-time entry bias reproduces call-frequency reduction but not the full joint call-recall / non-tool-final / invalid-call profile across seeds

Limitations

  • Single model family (Qwen3.5 at 4B and 9B); generalization to other architectures untested.
  • Only two-teacher routing studied; behavior with more teachers or more complex routing policies is unexplored.
  • Evaluated on three seeds; diagnostic evidence is observational (correlation between token-level signals and behavior), not a complete causal decomposition.
  • Targeted-clamp ablations rule out a first-token-only account but do not identify the exact responsible token positions.
  • A supervised format anchor (SFT on dataset-source rollouts, \(\alpha=0.3\)) is required for schema stability; pure teacher-distillation OPD without it is unstable.
  • No comparison with calibration methods outside the GKD family (e.g., RLHF-based or reward-model-guided approaches).

Relevance to Agentic AI / LLM Agents

Tool-call boundary control is a core reliability requirement for deployed LLM agents: over-calling raises latency and cost and can trigger runaway multi-turn tool-call loops. This paper offers a concrete, reproducible diagnosis of how a natural training strategy—multi-teacher distillation—silently corrupts the tool/no-tool decision boundary in ways aggregate losses obscure, directly implicating the on-policy distillation pipelines increasingly used to train agentic models. The behavior-leverage concept—that mode-entry tokens exert disproportionate trajectory control relative to their share of loss—is a transferable diagnostic lens for any agentic training setup with discrete mode boundaries (e.g., task-decompose vs. answer-directly, search vs. respond). Practically, Soft Clamp requires no architecture changes or inference-time modifications and plugs directly into existing GKD pipelines, lowering the barrier to adoption for researchers training tool-using agents.