Skip to content

Localizing RL-Induced Tool Use to a Single Crosscoder Feature

🕒 Published (v1): 2026-06-25 00:17 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML 2026; mechanistic localization of RL-induced tool use to single crosscoder feature

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RL fine-tuning for tool use concentrates its mechanistic signal into a surprisingly compact feature set that can be localized via Dedicated Feature Crosscoders (DFC). Steering a single DFC-exclusive feature achieves +65 pp tool-correctness on Qwen2.5-3B with zero retraining. Jointly training the crosscoder also leaks tool-calling capability into the frozen base model (+6.8 pp), revealing that RL-induced representations partially inhabit shared decoder directions.

Problem

RL post-training demonstrably improves agentic behaviors like structured tool-call generation, but the mechanistic basis of these changes is opaque: which internal features emerge, which are preserved, and whether those features can be leveraged for retraining-free behavioral control remain open questions. Prior Crosscoder work lacked systematic behavioral evaluation and had not demonstrated single-neuron steering saturation.

Method

The authors train 48 Dedicated Feature Crosscoder (DFC) variants on paired activations from a base model (Qwen2.5-3B) and its ToolRL-finetuned counterpart. A DFC partitions a shared sparse dictionary of size \(D\) into A-exclusive \([0, a_{end})\), B-exclusive \([a_{end}, b_{end})\), and shared \([b_{end}, D)\) sub-dictionaries via gradient masking, with training objective:

\[\mathcal{L} = \text{MSE}(h, \hat{h}) + \lambda_{sh}\|f_{sh}\|_1 + \frac{\lambda_{excl}}{2}\left(\|f_{A\cup sh}\|_1 + \|f_{B\cup sh}\|_1\right)\]

The 48-variant sweep covers architecture (DFC vs. unpartitioned CrossCoder), dictionary size \(D \in \{8192, 16384\}\), top-\(k \in \{45, 90, 160\}\), exclusive share \(p \in \{3\%, 5\%, 10\%\}\), and penalty \(\lambda_{excl} \in \{0, 10^{-3}\}\). Features are ranked by Cohen's \(d\) on tool-use vs. general-text activations and filtered by firing rate. Targeted steering applies an additive correction to the RL model's residual stream:

\[h'_A = h_A + \sum_{i \in S}(\alpha - 1) \cdot f_i \cdot W_{dec}[i, A, :]\]

Behavioral evaluation uses 100 held-out ToolRL prompts scored on format accuracy, tool correctness, and overall score.

Key Contributions

  • Systematic DFC sweep: First behavioral evaluation of DFC across 48 hyperparameter variants on a RL-finetuned model pair; encode-decode reconstruction improves RL-model tool correctness by \(+31.1 \pm 9.7\) pp.
  • Capability spillover: Passing the frozen base model's activations through the jointly-trained crosscoder passively transfers \(+6.8 \pm 5.0\) pp tool-correctness with zero fine-tuning; format accuracy never spills (0 pp), implying semantic tool-selection intent lives in shared directions but surface-form templating is RL-exclusive.
  • DFC as filter, not sink: Penalizing exclusive features (\(\lambda_{excl} = 10^{-3}\)) degrades RL-model fidelity, showing the exclusive partition concentrates but does not perfectly isolate RL-induced capability.
  • Single-neuron saturation: Steering one A-exclusive feature (\(|S|=1\), \(\alpha=32\)) achieves \(+65.0\) pp \(\Delta\) tool-correctness (95% CI \([+47.9, +82.1]\)), saturating performance across 7 of 9 probed layers; CrossCoder requires \(|S|=33\) to reach its unbudgeted peak.
  • Geometric validation: DFC decoder UMAP shows three spatially separated clusters (k-NN purity 0.984, HDBSCAN ARI 0.93); matched CrossCoder produces no separation (ARI 0.08, purity ~0.158 ≈ chance).

Results

  • Sweep mean reconstruction: base tool correctness 19% → 50.1% for RL model (\(+31.1 \pm 9.7\) pp); 0% → 6.8% for frozen base (capability spillover).
  • Single-feature steering (\(|S|=1\)): \(+65.0\) pp \(\Delta\) tool-correctness, Cohen's \(d_z = 1.03\), \(p = 2.2 \times 10^{-6}\).
  • CrossCoder unbudgeted ceiling: \(+70.0\) pp at \(|S|=33\); DFC vs. CC unbudgeted difference null (\(d_z = 0.00\), \(p = 0.51\)).
  • A-exclusive beats shared at \(|S| \leq 10\): \(+65.0\) pp vs. \(+47.5\) pp (\(d_z = 0.40\), \(p = 0.019\)); combo (A-excl ∪ shared) worse than A-excl alone (+35.0 pp, destructive interference).
  • Cross-layer generalization: mean best-cell \(+43.3\) pp across layers, paired-\(t\) \(p = 0.0006\); 2 of 9 layers dead (no tool-signed A-exclusive features).
  • Architecture comparison: CrossCoder \(+32.9\) pp / \(+8.9\) pp (A/B); DFC \(+30.5\) pp / \(+6.1\) pp; MSE indistinguishable (\(0.031\)); B-spillover difference directional but not significant (\(p = 0.12\)).

Limitations

  • Single model pair (Qwen2.5-3B) and single task (ToolRL); generalizability to other architectures or agentic tasks unstated.
  • Architecture comparison underpowered (12 CC vs. 36 DFC runs); directional DFC-reduces-spillover claim is not statistically significant.
  • Steering evaluated on only \(n=40\) prompts per cell; best-cell numbers not replicated on a fresh sample.
  • "Capability" operationalized as propensity to emit structured <tool_call> under fixed evaluation prompts; no prompted baseline maximally eliciting base-model tool use.
  • Autointerp pass covers only A-exclusive features; shared partition (which carries most spilled capability) not analyzed.
  • DFC partition is a filter, not a clean sink — RL capability partially leaks into shared directions, limiting the precision of feature-level isolation.

Relevance to Agentic AI / LLM Agents

This work directly addresses the mechanistic underpinnings of RL-trained agentic behaviors, showing that tool-use capability in a 3B-parameter model condenses into a handful of interpretable, steerable features — a result with concrete implications for runtime behavioral control of agents. The capability spillover finding is a security-relevant side effect: releasing crosscoders trained between base/RL pairs could inadvertently reintroduce RL-induced tool-use at inference time in nominally capability-limited models. For the broader line of work on model diffing and interpretability of post-training, this is the first study to demonstrate single-neuron steering saturation and to give behavioral rather than purely representational validation of DFC partitioning. The framing of RL-induced features as steerable objects — suppressing or amplifying specific agentic behaviors inference-time without retraining — positions mechanistic interpretability as a complement to RLHF/DPO for agent safety.