Skip to content

ToolWeaver: Weaving Collaborative Semantics for Scalable Tool Use in Large Language Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Collaborative tool semantics for scalable tool retrieval and use in LLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ToolWeaver replaces the "one-token-per-tool" paradigm in generative tool-use with a hierarchical compositional code representation, reducing vocabulary growth from linear to logarithmic in the number of tools. A collaborative-aware RQ-VAE tokenizer embeds both intrinsic semantics and extrinsic co-usage patterns into shared code prefixes, enabling LLMs to learn tool relationships from dense co-occurrence signals rather than sparse joint appearances of monolithic IDs. On ToolBench with ~47,000 APIs, ToolWeaver outperforms retrieval-based and generative baselines across tool retrieval and end-to-end task completion.

Problem

Existing generative tool-use methods assign each tool a unique special token ("one-token-per-tool"). This causes two compounding failures at scale: (1) scalability crisis — vocabulary size grows \(O(N)\), injecting ~47k out-of-vocabulary tokens that disrupt pretrained linguistic knowledge; (2) semantic bottleneck — flat, isolated token IDs prevent the model from learning collaborative relationships between tools, because the co-occurrence of any specific tool pair is statistically sparse in a large corpus.

Method

ToolWeaver represents each tool \(d\) as a compositional sequence of \(L\) discrete code indices \([\iota_1, \dots, \iota_L]\) drawn from \(L\) codebooks, each of size \(K\), yielding capacity \(K^L\) tools with only \(L \times K\) new tokens — a logarithmic expansion.

Structured tokenization proceeds in three steps: 1. A pretrained text encoder produces a dense semantic embedding \(e_d\) from tool documentation. 2. A collaborative-aware Residual Quantized VAE (RQ-VAE) sequentially quantizes residuals across \(L\) levels. At each level \(l\), \(\iota_{d,l} = \arg\min_k \|r_{d,l} - v_{l,k}\|_2^2\) and \(r_{d,l+1} = r_{d,l} - v_{l,\iota_{d,l}}\). A graph Laplacian regularization term $\(\mathcal{L}_{\text{collab}} = \sum_{u,v \in D} A_{uv}\|\hat{z}_u - \hat{z}_v\|_2^2\)$ penalizes distance between frequently co-occurring tools (similarity \(A_{uv}\) from co-occurrence counts), encouraging related tools to share early-level codes. 3. Conflict mitigation: a uniform mapping constraint on the final codebook \(C_L\) is solved as an optimal transport problem via the Sinkhorn-Knopp algorithm, ensuring unique per-tool identifiers without adding semantically vacuous tokens.

Generative alignment then fine-tunes the LLM in two stages: (1) query→code-sequence retrieval alignment; (2) full trajectory alignment (reasoning + tool calls + parameters). Constrained beam search with a trie of valid code sequences prevents hallucinated tool IDs at inference.

Key Contributions

  • Hierarchical compositional tool representation that scales vocabulary additions as \(O(L \cdot K)\) rather than \(O(N)\).
  • Collaborative-aware RQ-VAE tokenizer that injects co-usage signals via graph Laplacian regularization into the quantization objective.
  • Uniform mapping via optimal transport to eliminate index collisions without breaking semantic structure.
  • Two-stage generative alignment (retrieval + trajectory) integrating structured codes into an LLM.
  • Empirical validation on ToolBench (~47k APIs) showing state-of-the-art retrieval and end-to-end task completion.

Results

Tool retrieval (NDCG@k, ToolBench): - I1 (single-tool): ToolWeaver NDCG@1 = 91.16 vs. ToolGen 88.50, ToolRetriever 75.92. - I2 (multi-tool, same category): NDCG@1 = 89.76 vs. ToolGen 84.00, ToolRetriever 63.00. - I3 (cross-category, hardest): NDCG@1 = 88.00 vs. ToolGen 81.00, ToolRetriever 28.00.

End-to-end task completion (SoPR / SoWR, StableToolBench): - I3 SoPR: ToolWeaver 52.19 vs. ToolGen 36.34, GPT-4o-mini (retrieval) 24.86. - I3 SoWR: ToolWeaver 59.02 vs. ToolGen 49.18. - I1 SoPR: ToolWeaver 53.17 vs. ToolGen 52.97, GPT-4o-mini 52.25. - Unseen category generalization (I1-Cat SoPR): ToolWeaver 44.03 vs. ToolGen 45.13 (slight deficit) but I2-Cat SoPR 52.19 vs. ToolGen 36.34 (large gain). - Collaborative regularization weight \(\lambda\) peaks at 1; performance degrades at \(\lambda=10\).

Limitations

  • Requires pre-collected tool usage trajectories to build the co-occurrence matrix \(A\); cold-start tools with no usage history receive no collaborative signal.
  • The RQ-VAE and alignment fine-tuning add training complexity and cost compared to retrieval-only baselines.
  • Constrained beam search with a trie works only for closed, known tool sets; adding new tools mid-deployment requires re-tokenization and re-tuning.
  • The uniform mapping constraint is applied per-batch, so collision mitigation is probabilistic rather than guaranteed globally.
  • Generalization evaluation shows ToolWeaver slightly underperforms ToolGen on the I1-Cat. SoPR split (44.03 vs. 45.13), suggesting residual gaps for certain unseen-category scenarios.

Relevance to Agentic AI / LLM Agents

Scalable tool use is a foundational capability for autonomous LLM agents; ToolWeaver directly addresses the core bottleneck of deploying agents over realistically large API ecosystems (tens of thousands of tools). By encoding collaborative co-usage patterns into the tool representation itself, it advances agents' ability to reason about which tools work together — a prerequisite for multi-step planning tasks where no single tool suffices. The generative end-to-end paradigm (select + execute within one model) is architecturally cleaner than retrieval-augmented pipelines, reducing latency and error propagation in agentic loops. The logarithmic vocabulary scaling makes this approach viable as tool libraries continue to grow, a trend central to the current trajectory of tool-augmented agent research.