Skip to content

VLAS: Vision-Language-Action Model with Speech Instructions for Customized Robot Manipulation

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLAS is an end-to-end Vision-Language-Action model that directly ingests raw speech instructions for robot manipulation, eliminating the need for an external ASR system. It introduces a Voice Retrieval-Augmented Generation (RAG) module that uses speaker voiceprints to fetch individual-specific knowledge, enabling personalized task execution. A three-stage training curriculum and two new datasets (SQA, CSI) support speech-text alignment and manipulation fine-tuning.

Problem

Existing VLA models accept only text and image inputs; integrating speech via external ASR creates cascading pipelines that increase system complexity, add error propagation, and discard non-semantic speech cues (voiceprint, intonation, identity) that are essential for personalized, ambiguous instructions (e.g., "pick up my cup" when multiple cups are present).

Method

VLAS builds on LLaVA (CLIP visual encoder + Vicuna LLM backbone) and adds a Whisper speech encoder whose output (1500 hidden states, downsampled 5×) is projected into the shared language embedding space via a dedicated MLP. During inference, a Voice RAG module extracts a speaker voiceprint, queries a personal knowledge database, and injects the retrieved text as additional token context. Action output reuses 256 rarely-used LLM vocabulary tokens as discrete action bins across 7 DoF (Cartesian position + rotation + gripper). Training proceeds in three stages: (1) MLP-only speech-text alignment on LibriSpeech-360; (2) joint speech+VQA fine-tuning of all components except encoders using the new SQA dataset (185K image–audio pairs, 1,152 voices) and LLaVA's VQA data; (3) robot manipulation fine-tuning on the CSI dataset (23K CALVIN episodes, 194K audio instructions over 500 voices, with 50% of samples speech-replaced).

Key Contributions

  • First end-to-end VLA supporting raw speech instructions without an external ASR system.
  • Voice RAG paradigm: voiceprint-based speaker identification retrieves personal knowledge to resolve underspecified spoken commands.
  • VLAS-Base: a speech-extended LLaVA foundation model useful beyond robotics.
  • SQA dataset (185K image–speech QA pairs) and CSI dataset (23K robot episodes with speech instructions).
  • Customized-task benchmark covering object ownership, user preference, and compound multi-stage tasks.

Results

CALVIN benchmark (1000 long-horizon tasks, ABCD→D split): - VLAS+ (text): Len 3.74 vs. VLA baseline 3.80 — near-parity. - VLAS (TTS speech): Len 3.70 vs. VLA+ASR 3.13, Roboflamingo+ASR 3.41 — outperforms both cascaded baselines by large margins. - VLAS (real speech from 10 people): Len 3.61, gap of only 0.19 vs. text VLA baseline.

Customized-task benchmark (39 unseen voices): - VLAS (with Voice RAG): 86.5% average across ownership/preference/compound tasks. - VLA+ (text, no RAG): 19.2% average. - VLAS −RAG (ablation): 16.0% average — confirms RAG is essential. - VLA+ +RAG: 82.0% average — RAG benefit transfers to text-based VLA as well.

VLAS-Base multimodal benchmarks: - Matches LLaVA v1.5 on VizWiz (51.1 vs. 50.0), SQAI (72.2 vs. 66.8), GQA (62.0 vs. 62.0); no degradation from adding speech modality. - LibriSpeech WER: 2.79% vs. Whisper large-v2 2.7% — near state-of-the-art ASR. - SGQA (speech QA): 50.8% vs. BLIP-2 41.0% (using ground-truth text); lags LLaVA 62.0% (which sees ground-truth text).

Limitations

  • Speech data is TTS-synthesized (ESPnet/VITS on LibriTTS); real speech instructions show a performance drop (~5–7 pp on customized tasks), indicating a sim-to-real gap.
  • The 5× downsampling of speech tokens reduces WER accuracy; no optimal factor was explored.
  • Voice RAG requires pre-populated personal knowledge databases and a reliable speaker identification module; cold-start users or noisy voiceprints are not evaluated.
  • Only tested on CALVIN simulation and a limited real-world cup-picking task with a UR5 arm; broader manipulation generalization is untested.
  • VLAS-Base still lags LLaVA on speech QA (SGQA 50.8 vs. 62.0) because it receives speech rather than ground-truth text.

Relevance to Vision-Language Models

VLAS demonstrates a practical pathway to extend VLMs beyond text+image inputs to include raw speech, without sacrificing multimodal QA performance — a direct extension of the VLM modality-expansion line (e.g., ImageBind-LLM, VITA). The Voice RAG mechanism is an instance of knowledge-augmented VLM inference that exploits modality-specific metadata (voiceprint) rather than textual queries, a novel coupling of speaker diarization with retrieval-augmented generation for grounded decision-making. For VLM researchers, VLAS-Base itself is a released speech-capable LLaVA variant suitable for non-robotic downstream tasks. The work also highlights that non-semantic speech cues carry task-critical information that text transcription discards, a consideration relevant to any multimodal system that converts speech to text before VLM processing.