Skip to content

Inner Speech as Behavior Guides: Steerable Imitation of Diverse Behaviors for Human-AI coordination

šŸ•’ Published (v1): 2025-01-01 Ā· Source: NeurIPS Ā· Venue: NeurIPS 2025 Ā· link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

šŸ’¬ Ask ChatGPTāœ¦ Ask Claude

TL;DR

MIMIC (Modeling Inner Motivations for Imitation and Control) uses vision-language models as off-line linguistic scaffolding to train a conditional variational autoencoder (CVAE) that generates natural-language "inner speech" from behavioral history, which then conditions a diffusion-based behavior cloning policy. This mediational architecture captures behavioral diversity and enables inference-time steering without additional demonstrations, grounded in Vygotsky's theory of inner speech as a cognitive mediator.

Problem

Standard behavior cloning treats human action as a direct state-to-action mapping, missing the non-Markovian, multimodal nature of human behavior and providing no mechanism for controlled generation at inference time. Methods requiring explicit human thought annotation per step (e.g., Thought Cloning) are unscalable, and goal-conditional approaches offer only coarse control.

Method

MIMIC decomposes the action distribution through an inner speech latent mediator: p(a|s) = āˆ« p(a|s,m)Ā·p(m|s)dm.

Three components: 1. VLM scaffolding (training only): GIFs of each demonstration are passed to a VLM (default: GPT-4o) with a discriminative prompt asking it to generate inner thoughts distinguishing behaviors across a batch of 8 GIFs. The resulting natural-language descriptions are embedded via CLIP to obtain inner speech vectors mā½ā±ā¾. 2. Behavior-conditioned inner speech generator: A CVAE (encoder + decoder conditioned on a window W of past images) is trained to reconstruct the VLM-generated CLIP embeddings, with a Ī²-annealed KL loss enforcing semantic condensation. At inference, the CVAE decoder generates inner speech every W steps from accumulated observation history. 3. Inner-speech-conditioned behavior cloner: A DDPM-T (diffusion policy with transformer) trained with classifier-free guidance to predict actions given (state, inner speech). At inference time, a designer-specified text description overrides the initial inner speech, and the CVAE then continues periodic updates to maintain consistency.

Key Contributions

  • Theoretical formalization of inner speech as a latent mediator in imitation learning, grounded in Vygotsky's predicativity, semantic condensation, and temporal regulatory dynamics.
  • MIMIC framework: VLM scaffolding ā†’ CVAE inner speech generator ā†’ DDPM-T policy, trained without any per-step human thought annotation.
  • Inference-time behavioral steering via natural language descriptions, without retraining.
  • Empirical demonstration that inner speech improves both behavioral diversity (entropy) and fidelity (success rate, Wasserstein distance) on D3IL robotic benchmarks and collective reward in Overcooked human-AI coordination.

Results

  • Aligning (D3IL): MIMIC-S: success rate 0.80 vs. BC 0.66; state Wasserstein 0.046 vs. BC 0.696; time Wasserstein 50.6 vs. BC 59.0.
  • Sorting-Vision: MIMIC-E entropy 0.449 vs. BC 0.360; success rate 0.808 vs. 0.797.
  • Stacking (2 boxes): MIMIC-S success 0.607 vs. BC 0.488; entropy substantially higher.
  • Overcooked (Cramped room): MIMIC collective reward 151.8 Ā± 2.45 vs. BC 115.8 Ā± 3.86 (+31%).
  • Overcooked (Cramped room-Vision): MIMIC 108.8 Ā± 4.84 vs. BC 73.6 Ā± 6.18 (+48%).
  • VLM comparison: GPT-4o best success rate; Qwen2.5-VL-72B close second; o4-mini highest entropy but lower success (CLIP embedding mismatch with nuanced descriptions).
  • Ablation: Language-based inner speech (MIMIC) > K-means cluster vectors > random vectors > base BC, confirming linguistic representation is key.

Limitations

  • VLM inference for GIF-based description generation is run only at training time, but still incurs compute cost proportional to dataset size; larger datasets may be expensive.
  • Performance depends on VLM quality and CLIP's ability to distinguish the generated descriptions (o4-mini descriptions are too nuanced for CLIP).
  • Vision-based environments show smaller absolute gains (e.g., Aligning-Vision: 0.207ā€“0.223 vs. BC 0.183), suggesting visual encoding is still a bottleneck.
  • Non-Markovian history window W is a hyperparameter requiring per-dataset tuning.
  • Evaluated only on relatively constrained tabletop manipulation and 2-player cooperative tasks; generalization to richer real-world settings is unvalidated.
  • No explicit mechanism to handle distribution shift beyond behavior cloning's known brittleness, though periodic inner speech updates mitigate it partially.

Relevance to Vision-Language Models

VLMs (GPT-4o, Qwen2.5-VL-72B, o4-mini) serve as zero-shot behavioral annotatorsā€”converting raw image sequences into discriminative natural-language descriptions that become supervision targets for a lightweight CVAEā€”demonstrating a scalable, annotation-free pipeline that repurposes VLM visual grounding for downstream imitation learning. The work reveals a concrete sensitivity: the choice of VLM and downstream text encoder (CLIP vs. MPNET) jointly determine behavioral diversity and policy fidelity, making VLM-to-embedding compatibility a first-class design constraint. For VLM researchers, this frames VLMs as "behavior linguists" whose description quality and stylistic diversity directly govern agent expressiveness, opening a new evaluation axis beyond standard captioning or QA benchmarks. It also contributes a methodologyā€”batch-contrastive prompting over GIFsā€”for eliciting behaviorally discriminative language from VLMs without labeled annotations.