Skip to content

Multi-Head Latent Control: A Unified Interface for LLM Agent Decision Making

🕒 Published (v1): 2026-07-15 18:36 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Multi-Head Latent Control (MHLC) attaches two lightweight, post-hoc classifier heads to a frozen LLM or VLM to produce deployment-time control signals: a Capability Head that decides whether to escalate an instance to a stronger fallback model, and a Resolution Head that selects among clarification, tool invocation, abstention, or direct answering. Trained solely on hidden-state traces from the frozen backbone, MHLC reduces large-model API cost by 27–90.7% while preserving most large-model performance, and improves tool-use decision quality by up to +158.9% relative score on TriviaQA.

Problem

Existing agentic deployment strategies—prompt-level routing, external orchestration, and task-specific fine-tuning—rely on input-side signals and require re-training whenever a new backbone is released. They do not address instance-level adequacy estimation at inference time, leaving substantial compute waste: stronger models are invoked even for easy instances, and intervention decisions (when to call a tool, when to abstain) are delegated to the backbone's surface behavior rather than its internal representational state.

Method

MHLC adds two independent linear heads on top of a frozen backbone (LLM or VLM); the backbone is never modified.

Latent representations. For an input \(x\) generating output tokens \((\hat{y}_1, \ldots, \hat{y}_N)\), the Capability Head reads the final-layer hidden-state trace \(H^{(L)} \in \mathbb{R}^{N \times d}\), while the Resolution Head reads a selected mid-layer trace \(H^{(\ell_{\text{res}})}\). Each trace is compressed to a fixed-budget representation via a pooling encoder \(\Pi(\cdot)\).

Capability Head. Predicts a scalar adequacy score \(p_{\text{cap}} = \sigma(h_{\text{cap}}(\mathbf{z}_{\text{cap}})) \in [0,1]\) using weighted MSE against LLM-judge-assigned scalar labels. If \(p_{\text{cap}} < \tau_{\text{cap}}\) (default 0.8), control transfers to the stronger model \(m_2\).

Resolution Head. Conditioned on retaining control, predicts a three-vector \(\mathbf{s}_{\text{res}} = [s_{\text{info}}, s_{\text{tool}}, s_{\text{cant}}] \in [0,1]^3\) via independent binary cross-entropy over action classes \(\mathcal{A} = \{\text{info, tool, cant}\}\); direct answering is the implicit all-zero state. An intervention fires only when \(\max_a s_{\text{res},a} > \tau_{\text{res}}\) (default 0.5).

Training. Generates rollouts with the frozen backbone, collects hidden-state traces, and derives supervision from an LLM judge (Capability) or ground-truth resolution labels (Resolution Head, using the When2Call dataset). Training for both heads on a 9B backbone completes in under one day on a single GPU and fits in 16 GB VRAM.

Key Contributions

  • Post-hoc control layer for frozen LLMs/VLMs that requires no backbone fine-tuning and adapts rapidly to new model releases.
  • Capability Head: instance-level adequacy signal from final-layer hidden states enabling selective escalation to a stronger model.
  • Resolution Head: structured intervention prediction (clarification / tool use / abstention / direct answer) from mid-layer hidden states.
  • Prefix-time adequacy estimation: the Capability Head reliably predicts adequacy from partial generations, enabling early handoff before full decoding.
  • Unified control interface: a single latent reading supports model selection and intervention decisions across text-only and vision-language modalities.

Results

Capability-guided routing (Table 2, across Qwen3-VL, Qwen3.5, Gemma families): - Overall paid API cost reduction: 27–53% on average across six benchmarks relative to always-using-large-model, while matching or nearly matching large-model task scores. - Routed Qwen3.5-9B→27B-Thk: 53.0% overall cost reduction with equivalent average score to \(m_2\). - Routed Qwen3-VL-4B-Thk→32B-Thk: 54.4% overall cost reduction.

AndroidWorld long-horizon agentic setting (Table 1): - Qwen3-VL-4B→32B routed: success rate 0.60 vs. 0.58 for \(m_2\) alone, API cost $0.25 vs. $2.70 — a 90.7% cost reduction. - Qwen3.5-9B→27B routed: success rate 0.56 vs. 0.59, 85.8% cost reduction ($0.21 vs. $1.49).

Resolution Head on When2Call (Table 3, referenced but not shown in excerpt): - Up to +11.7 F1 points and +12.4 accuracy points over backbone-native intervention decisions.

Tool-use decisions on TriviaQA (§4.3): - Up to +158.9% relative score gain and 65.5% fewer missed required tool calls compared to backbone-only tool-calling behavior.

Limitations

  • Resolution Head training is tied to the When2Call dataset; generalization to intervention taxonomies beyond {clarification, tool use, abstention} is not demonstrated.
  • Layer selection for each head is determined empirically per backbone; there is no principled rule for choosing \(\ell_{\text{res}}\) across architectures.
  • Scalar adequacy labels are generated by an LLM judge (Qwen3-VL-30B-A3B), introducing judge-induced noise and a potential circularity when the judge and backbone are from the same family.
  • The method assumes access to hidden states, precluding use with black-box API-only models.
  • Cost estimates are token-based proxies; latency costs of running the small model first and then potentially the large model are not reported.
  • Capability Head performance on fully text-only benchmarks (MMLU-Pro, TriviaQA) is shown, but backbone families are limited to Qwen and Gemma; broader generalization is untested.

Relevance to Vision-Language Models

MHLC is directly evaluated on the Qwen3-VL family (2B, 4B, 32B, including thinking variants), making its hidden-state control framework natively applicable to VLMs without architectural modification—the method excludes vision prompt tokens and reads only generated-token hidden states, providing a modality-agnostic interface. For researchers tracking VLMs, the Capability Head demonstrates that VLM hidden states contain reliable adequacy signals that transfer across tasks spanning multimodal QA, GUI grounding (ScreenSpot-Pro), chart reasoning (CharXiv), and agentic mobile control (AndroidWorld). The work connects to the broader VLM deployment challenge of balancing capability against inference cost, complementing model compression and speculative decoding by instead controlling when a large VLM is invoked.