Controlling Tool Use with Heading-Specific Activation Steering

🕒 Published (v1): 2026-07-07 03:34 UTC · Source: Arxiv · link

Why this paper was selected

Activation steering for tool-use decisions; mechanistic handle to suppress spurious tool calls

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper investigates whether LLM tool-use decisions have stable internal representations that can be steered at inference time, despite tools having no parametric encoding in model weights. By extracting steering vectors from hidden states at structured heading-anchor positions (e.g., ### Code), the authors achieve bidirectional causal control over tool invocation across five open-source models. Suppression is effective in parametric-reasoning domains but degrades where external retrieval is genuinely necessary, and geometric analysis reveals the underlying representations are non-linear and diffuse rather than cleanly encoded.

Problem

Tool-augmented LLMs suffer from tool overuse: invoking external tools when parametric reasoning would suffice, causing unnecessary latency, cost, noisy retrieval, and error propagation. Existing fixes either require expensive retraining or operate at the output interface rather than on internal activations. More fundamentally, it is unclear whether tool-use decisions — which are entirely context-dependent and lack any encoding in model weights — have extractable internal representations at all, making standard activation-steering assumptions inapplicable.

Method

The approach uses heading-specific contrastive activation addition. The model is prompted to emit structured intermediate reasoning with explicit heading anchors (### Reasoning, ### Search, ### Code, ### AskUser). Steering vectors are extracted per tool type \(k\) using 60 labeled trajectories (20 per domain) generated by GPT-4o.

For each trajectory \(i\) and layer \(l\), the method averages hidden states at all ### Reasoning anchor positions (\(\bar{h}^{(l)}_{R,i}\)) and at all target tool heading positions (\(\bar{h}^{(l)}_{k,i}\)), then computes a per-trajectory delta:

\[\Delta^{(l)}_{k,i} = \bar{h}^{(l)}_{R,i} - \bar{h}^{(l)}_{k,i}\]

The steering vector is averaged across \(N_k\) trajectories:

\[v^{(l)}_k = \frac{1}{N_k}\sum_{i=1}^{N_k} \Delta^{(l)}_{k,i}\]

At inference time, the intervention fires only at heading-formation tokens (###). Activation addition subtracts the vector from the residual stream (coefficient \(\alpha=1\)) to suppress tool use:

\[\tilde{h}^{(l^*)}_t = h^{(l^*)}_t + \alpha v^{(l^*)}_k\]

Orthogonalization removes the suppressive component to amplify tool use:

\[\tilde{h}^{(l^*)}_t = h^{(l^*)}_t - \text{proj}_{v^{(l^*)}_k}(h^{(l^*)}_t)\]

Steering layer \(l^*\) is selected per model by sweeping all layers and picking the elbow of the tool-usage curve; all five models' elbows fall at normalized depth 0.45–0.60.

Key Contributions

• Demonstrates that inference-time activation steering can bidirectionally control tool invocation in tool-augmented LLMs without any parameter updates, despite tools having no parametric weight encoding.
• Introduces heading-specific steering vector extraction anchored at structured output tokens, enabling lightweight, localized intervention at heading-formation steps.
• Shows domain-asymmetric suppression: near-zero tool use on Math (parametric) with modest accuracy loss, but larger degradation on Time and Intention where retrieval/clarification is genuinely required.
• Reveals that all five models' effective steering layers cluster at normalized depth 0.45–0.60, suggesting consistent relative depth of tool-use representations across architectures.
• Provides geometric analysis showing that causal effectiveness does not require clean linear encoding: tool-step hidden states exhibit diffuse bimodal cosine similarity with the steering vector, distinguishing this from parametrically grounded concepts like sentiment or refusal.
• Demonstrates partial cross-format transfer (Markdown-extracted vector suppresses 63.9% of tool calls in a JSON schema) while characterizing the direction as schema-sensitive rather than fully format-independent.

Results

• Math (Code steering): Activation addition reduces ToolAvgUse from 3.90→0.56 (Mistral-7B), 1.93→0.35 (Llama-3.1-8B), 1.55→0.13 (Mistral-Small) with only modest ACC drops, suggesting many calls are redundant.
• Time (Search steering): Larger accuracy penalties; Llama-3.1-8B drops 56.0→45.0 ACC as ToolAvgUse falls 2.05→1.50; Mistral-Small drops 62.0→49.0 ACC.
• Intention (AskUser steering): Missing Details Recovery collapses across all models (e.g., Mistral-7B: 48.84/21.70 → 14.3/4.67), confirming these calls are necessary.
• Orthogonalization consistently increases tool usage above baseline across all 5 models and 3 domains, but does not improve and often degrades task accuracy.
• Larger models show greater resistance: Llama-3.1-70B retains more tool calls under suppression than smaller models.
• Cross-format control: Markdown-to-JSON transfer suppresses 63.9% of tool calls with near-zero accuracy change (0.330→0.325); heading rename retains 70.3% relative suppression.
• Linear probe vs. mean-difference: On Llama-3.1-70B a linear probe achieves AUROC 1.000 separating heading types, but the mean-difference vector is far more causally effective (ToolAvgUse 0.235 vs. 1.235), showing linear decodability and causal effectiveness are dissociable.

Limitations

• Evaluated on only three domains from the SMART benchmark; generalization to arbitrary tool ecosystems is unverified.
• Steering vectors require 60 labeled trajectories generated by GPT-4o, introducing a data dependency and potential distributional mismatch.
• Cross-format transfer is preliminary: single-model (Llama-3.1-8B), single-domain (Math) controls; reverse transfer (JSON→Markdown) is substantially weaker (15.5%).
• The disconnect between geometric irregularity and causal effectiveness is stated as an open question; no mechanistic account reconciles them.
• Multi-turn dependency structure prevents direct comparison between unsteered geometric distributions and steered trajectories (the probing and steering analyses are on different trajectory distributions).
• Method requires structured heading-anchor prompting; applicability to models not formatted this way is unclear.
• Larger models (Llama-3.1-70B) show greater resistance to suppression, suggesting scalability challenges.

Relevance to Agentic AI / LLM Agents

This work directly addresses a core failure mode of LLM agent systems — tool overuse — through inference-time representation manipulation, offering a parameter-free alternative to retraining-based solutions. It provides mechanistic insight into where in the network tool-invocation decisions are encoded (middle layers, normalized depth 0.45–0.60) and challenges the assumption that activation steering requires parametric concept grounding, which has implications for steering any agentic behavior driven by context rather than weight-encoded knowledge. The heading-anchor methodology is transferable to any structured agent scaffolding that uses delimited action tokens, and the domain-asymmetric suppression results constitute a principled empirical test for when parametric reasoning can substitute for tool access. The geometric finding that causal effectiveness and linear encoding are dissociable is a broader result for the representation engineering literature applied to agentic settings.