Interleave-VLA: Enhancing Robot Manipulation with Image-Text Interleaved Instructions

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; interleaved image-text instructions improve robot manipulation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Interleave-VLA extends text-only Vision-Language-Action models to accept interleaved image-text instructions, enabling in-context visual grounding for robot manipulation. It contributes an automatically generated dataset of 210k real-world interleaved episodes from Open X-Embodiment and achieves roughly 2× better out-of-domain generalization over text-only baselines. The paradigm is model-agnostic and requires only a lightweight tokenizer adaptation.

Problem

Existing VLA models are conditioned on text-only instructions, which are ambiguous in cluttered scenes and carry distributional biases from training data. These issues cause three failure modes the authors term attentional hallucinations: (1) attentional bias — focus on prominent distractors instead of the target; (2) diffused attention — no clear focal point due to model uncertainty; (3) attention leakage — correct target identified but attention spreads to irrelevant background regions. No large-scale real-world interleaved embodied dataset existed to train policies beyond this paradigm.

Method

Paradigm. The policy \(\pi_\theta\) generates actions \(a_t \sim \pi_\theta(\cdot \mid s_t)\) where the state \(s_t = (I_t, q_t, \mathcal{I})\) includes a current visual observation \(I_t\), proprioceptive state \(q_t\), and interleaved instruction \(\mathcal{I} = (u_1, \ldots, u_M)\) with \(u_j \in \mathcal{V}_\text{text} \cup \mathcal{V}_\text{img}\). Text-VLA is a degenerate case where all \(u_j \in \mathcal{V}_\text{text}\).

Adaptation. Special separator tokens (<BOI>, <EOI>) are injected into the base VLA tokenizer (applied to π0 / PaliGemma backbone), enabling interleaved input processing without changing the core architecture or flow-matching objective.

Dataset construction pipeline for the Open Interleaved X-Embodiment Dataset: 1. Instruction parsing: Qwen2.5 extracts key object nouns from text instructions across diverse formats. 2. Open-vocabulary detection: OWLv2 locates and crops target objects from trajectory frames; achieves 82.6% accuracy. 3. Quality verification: Qwen2.5-VL verifies detections; failures are corrected with keypoint-guided Segment Anything, raising accuracy to 95.6%.

The resulting dataset integrates 11 OXE sub-datasets (RT-1, Bridge, BC-Z, etc.), totalling 210k episodes, 13M frames, and 3,500 unique objects. Internet-sourced images are randomly mixed in to increase visual heterogeneity.

Inference supports camera crops, web images, and hand-drawn sketches as instruction images, enabling zero-shot generalization to novel instruction styles unseen during training.

Key Contributions

• Interleave-VLA paradigm: lightweight, model-agnostic plug-in that adds interleaved image-text instruction support to existing VLAs with minimal modification.
• Open Interleaved X-Embodiment Dataset: 210k episodes / 13M frames, fully automatically generated via a VLM-assisted pipeline; released open-source.
• Attentional hallucination taxonomy: characterizes and quantitatively analyzes three failure modes of Text-VLA (attentional bias, diffused attention, attention leakage) and demonstrates that interleaved visual grounding mitigates them.
• Emergent zero-shot capabilities: the trained model generalizes to hand-drawn sketches and web images not present in training data.

Results

• SimplerEnv simulation (π0 base model):
• In-domain: Text-VLA 68.1% vs. Interleave-VLA Full 70.5% (on par)
• Novel Object (OOD): Text-VLA 26.0% → Interleave-VLA Full 53.8% (+27.8 pp, ~2×)
• Novel Category (OOD): Text-VLA 19.3% → Interleave-VLA Full 57.3% (+38.0 pp, ~3×)
• Average OOD: Text-VLA 39.5% → Interleave-VLA Full 60.6% (+21.1 pp)
• vs. π0.5 (which uses additional object-grounding VQA pretraining): Interleave-VLA Full exceeds π0.5 (49.0% avg OOD) by 11.6 pp.
• Real-robot experiments: gains of +29.4, +37.0, +30.2, +40.1, +32.0, +12.6, +29.4, +1.8 percentage points across reported tasks over Text-VLA baseline (Figure 1b).
• Interleave-VLA (Partial) — trained on interleaved data but tested with text — already outperforms Text-VLA on OOD tasks, confirming cross-modal training benefits even without image-conditioned inference.

Limitations

• Full generalization gains require image instructions at test time; users must supply crop, web image, or sketch, which adds interaction overhead.
• Automatic dataset labeling reaches only 95.6% accuracy; ~4.4% of interleaved labels are noisy.
• Evaluation is confined to tabletop manipulation (BridgeData / SimplerEnv-Bridge, FANUC arm); generalization to locomotion, navigation, or dexterous manipulation is untested.
• The backbone (π0 / PaliGemma) is not natively pre-trained on interleaved data, so the adaptation relies on fine-tuning to activate capabilities the base model may possess only weakly.
• Internet-image augmentation introduces distribution shift; whether this helps in high-precision tasks is not ablated.

Relevance to Vision-Language Models

This work directly operationalizes the interleaved VLM paradigm (Flamingo, Qwen-VL, etc.) in the action domain, showing that VLM-style multimodal inputs transfer their generalization benefits to embodied control — closing a gap between digital-world VLMs and physical-world VLAs. The attentional hallucination analysis is directly relevant to the broader VLM hallucination literature, providing an embodied-AI lens on why image grounding reduces object confusability in a way that text tokens alone cannot. The dataset construction pipeline is itself a showcase of VLM tool-chaining (Qwen2.5 + OWLv2 + Qwen2.5-VL + SAM) for scalable annotation, a pattern of growing relevance to multimodal data curation. For VLM researchers, the result that cross-modal training on heterogeneous interleaved data improves even text-only evaluation performance (Interleave-VLA Partial) reinforces the value of diverse modality exposure during training.