Skip to content

Multimodal Policy Internalization for Conversational Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Multimodal policy internalization for conversational agents; reduces explicit policy reliance

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Multimodal Policy Internalization (MPI) is a new task and training framework for embedding complex, reasoning-intensive policies (tool-use rules, decision trees) into multimodal model weights so those policies need not appear in-context at inference. The proposed three-stage TriMPI pipeline — visually-masked continual pretraining → CoT SFT → RL with PolicyRollout — achieves up to 70.7 percentage-point absolute gains over CoT-SFT baselines while reducing inference token cost by 85–94%.

Problem

Production conversational agents carry system prompts estimated at 1K–50K tokens, creating a 20×–250× input-token overhead relative to typical 50–200 token user queries. Models also struggle to faithfully follow long, reasoning-intensive policies in-context. Prior work on prompt compression addressed only template/demonstration reduction requiring minimal reasoning; deliberative alignment internalized text-only safety rules. No prior work studied multimodal policies that include visual instructions (e.g., demo images as decision-node conditions) or complex tool-calling rules dependent on user attributes.

Method

TriMPI is a three-stage training framework operating on a multimodal model \(M_\theta\):

  1. VM-CPT (Visually-Masked Continual Pretraining): Standard CPT loss over sequences \((P_T, P_I, I, Q, C, A)\) but with a mask \(m_t = \mathbf{1}[x_t \notin P_I \cup I]\) that excludes continuous visual tokens from the loss. This directly injects policy knowledge into \(\theta\) before any task-specific training.

  2. CoT SFT: Supervised fine-tuning on chain-of-thought data where the model reasons through policy rules before answering, without the policy in the prompt.

  3. RL with PolicyRollout (PoRo): Extends GRPO by augmenting each rollout batch: for every instance \((Q, I)\), an additional set of responses is sampled conditioned on \((Q, I, P)\) (policy in-context). Both sets are pooled for group-advantage estimation, but the policy gradient is applied only to the no-policy path (\(r_i(\theta) = \pi_\theta(o_i|Q,I)/\pi_{\theta_\text{old}}(o_i|Q,I)\)), preserving train–inference alignment:

\[J_\text{PoRo-GRPO}(\theta) = \mathbb{E}_{\{o_i\}_{i=1}^G \sim \pi_{\theta_\text{old}}(\cdot|Q,I),\; \{o_j\}_{j=G}^{2G} \sim \pi_{\theta_\text{old}}(\cdot|Q,I,P)} \frac{1}{2G}\sum_{i=1}^{2G}\left[\min\!\left(r_i(\theta)\hat{A}_i,\,\text{clip}(r_i(\theta),1\!-\!\epsilon_l,1\!+\!\epsilon_h)\hat{A}_i\right) - \beta D_\text{KL}[\pi_\theta\|\pi_\text{ref}]\right]\]

Two datasets support evaluation: ClevrPolicy (synthetic CLEVR images; binary decision-tree policies of varying depth N=2/4/6; text-only ClevrPolicy-T and multimodal ClevrPolicy-M variants) and GTAPolicy (real-world images; 13 tool types, 24 user-conditional tool-calling rules; low-data regime).

Key Contributions

  • Formal definition of the Multimodal Policy Internalization (MPI) task: train \(M_\theta\) so \(A = M_\theta(Q,I)\) matches behavior of \(M_\theta(Q,I,P)\).
  • ClevrPolicy dataset: controllable decision-tree complexity, both text-only and multimodal policy variants.
  • GTAPolicy dataset: real-world tool-use with user-conditional routing rules, low-data regime.
  • TriMPI framework combining VM-CPT, CoT SFT, and RL in sequence.
  • PolicyRollout (PoRo): policy-aware rollout augmentation for GRPO/DAPO that avoids train–inference distribution mismatch.

Results

  • TriMPI w/ PoRo-DAPO vs. CoT SFT baseline (Qwen2.5-VL-7B, N=6 policies):
  • ClevrPolicy-T: 77.80% vs. 14.55% (+63.25 pp)
  • ClevrPolicy-M: 85.00% vs. 14.30% (+70.70 pp)
  • GTAPolicy Overall: 76.01% vs. 40.75% (+35.26 pp)
  • vs. in-context (Qwen2.5-VL-7B zero-shot):
  • ClevrPolicy-T N=6: 77.80% vs. 13.15% (+64.65 pp)
  • GTAPolicy Overall: 76.01% vs. 21.51% (+54.50 pp — up to 79.4 pp across conditions)
  • PolicyRollout ablation (PoRo-DAPO vs. DAPO without PoRo): ClevrPolicy-T 77.80% vs. 65.85%; ClevrPolicy-M 85.00% vs. 81.45%.
  • VM-CPT ablation: removing VM-CPT (PoRo-GRPO without CPT) drops ClevrPolicy-T from 65.85% to 47.05%.
  • Inference efficiency: 85.7%–93.9% reduction in input tokens after internalization.
  • Even Claude-4-Sonnet in-context achieves only 55.96% overall on GTAPolicy and 90.10% on ClevrPolicy-T N=6, showing dataset difficulty.

Limitations

  • VM-CPT requires full parameter tuning (no LoRA), making it computationally expensive; PoRo doubles rollout generation cost during RL.
  • GTAPolicy operates in a strict low-data regime, which limits RL exploration and may cause DAPO overfitting.
  • Evaluations use Qwen2.5-VL-7B as primary base; generalization to other architectures or much larger models is not demonstrated.
  • The policy must be fixed at training time; dynamic policy updates require retraining or override mechanisms (partially addressed in generalization experiments but at reduced performance relative to full retraining).
  • Visual masking in VM-CPT discards all visual token loss signal, which may limit learning from visual policy demonstrations.

Relevance to Agentic AI / LLM Agents

This work directly attacks a core bottleneck in production agentic systems: the token cost and reliability of long system-prompt policies governing tool selection, user routing, and behavior rules. By internalizing these policies into weights, MPI enables agents with the same behavioral specification to operate at a fraction of the inference cost — critical for high-throughput deployments. PolicyRollout introduces a principled method for RL training when inference-time context differs from training context, a generalizable pattern applicable to any agent that reasons under policies it cannot access at test time. The work extends deliberative alignment into the multimodal domain and connects tightly to tool-use learning and RLVR for agents.