Skip to content

LLaRA: Supercharging Robot Learning Data for Vision-Language Policy

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LLaRA converts existing behavior-cloning (BC) robot demonstration datasets into conversation-style visual instruction-tuning data, then finetuning a pretrained VLM (LLaVA-1.5) on this data to produce a Vision-Language-Action (VLA) model. Robot actions are expressed as 2D image pixel coordinates in natural language rather than special quantized tokens, enabling efficient VLM transfer. Six self-supervised auxiliary tasks (localization, detection, action prediction, future prediction, spatial/temporal relationships) further boost performance without additional action annotations.

Problem

Directly adapting pretrained VLMs for low-level robotic control is data-inefficient: prior VLA approaches (RT-2, OpenVLA) encode actions as special quantized tokens that are semantically foreign to the VLM's pretraining distribution, requiring large amounts of robot demonstrations to learn. There is also a lack of principled methods for generating self-supervised auxiliary data from existing robot trajectories to augment limited BC datasets.

Method

Visuomotor Instruction Tuning reformulates BC datasets as single-round question-answer conversations (Instruct-BC / inBC): the visual observation and task description form the user query; the robot action is the response, expressed as normalized 2D image coordinates and rotation angles in natural language (e.g., pick up <p>obj</p> at <b>(0.262, 0.668)</b>, rotate <r>[-148]</r> degrees). A variant, D-inBC, converts multimodal task reference images into textual object-detection descriptions to handle single-image VLMs.

On top of inBC/D-inBC, six auxiliary datasets are generated automatically from the same trajectories via scripted templates (rephrased with GPT-4 for diversity): 1. Localization — query: object name → response: bounding box (x,y), {w,h} 2. Detection — full scene object list 3. Action prediction — given two observations, identify the intervening action 4. Future prediction — given current observation + action, describe the next state 5. Spatial relationship — 2D/Euclidean distance and direction between object pairs 6. Temporal relationship — how spatial relationships change across timesteps

The backbone is LLaVA-1.5 (single-image VLM + CLIP encoder + MLP adapter + LLM); only θ_MLP and θ_LLM are finetuned in a single stage.

Key Contributions

  • Formulation of robot manipulation as natural-language instruction-response pairs aligned to image pixel coordinates, enabling direct reuse of VLM conversational pretraining.
  • Automated pipeline (inBC / D-inBC) to convert any BC dataset into VLM-compatible visuomotor instruction data without architecture modifications.
  • Six self-supervised auxiliary instruction-tuning datasets generated from existing trajectories with no extra action labels.
  • Empirical demonstration that LLaRA with only 12% of VIMA training data outperforms VIMA-200M trained on 100% of data.

Results

  • VIMA-Bench (simulation): LLaRA (D-inBC + Aux(B) + Oracle, trained on 12% / 8k trajectories) achieves L1=90.0%, L2=88.1%, L3=79.2% vs. VIMA-200M (100% data): L1=80.7%, L2=81.9%, L3=77.9%.
  • inBC vs RT-2 Style (VIMA-8k): inBC scores L1=57.3%, L2=46.2%, L3=42.9% vs. RT-2 Style L1=3.8%, L2=3.1%, L3=1.7%; adding Aux(D) to inBC: L1=59.2%, L2=58.8%, L3=52.1%.
  • RT-2 Style + Aux(D) (VIMA-8k): dramatically improves to L1=36.9%, L2=35.8%, L3=32.1%, confirming auxiliary data benefits even non-coordinate methods.
  • Zero-shot real-world (ZS): inBC achieves 36.6% avg. vs. GPT-4o 31.6%, RT-2 Style (VIMA-8k) 0%, RT-2 Style (VIMA-660k) 0%.
  • Real-world finetuning (FT): D-inBC + Aux(C) + OD reaches 91.6% avg. (T1=90%, T2=100%, T3=85%) with xArm-Action data.
  • Real-world joint training (JT): D-inBC + Aux(C) + OD achieves 83.3% avg. with xArm-Det.
  • Generalization: 75% success on T1 with distractors (unseen), 40% on T1 with complex rephrasing, after finetuning on xArm-Action.

Limitations

  • Relies on predefined calibration to map 2D image coordinates to 3D robot action space; does not generalize to freely-moving cameras or non-planar workspaces without recalibration.
  • Auxiliary dataset benefits diminish with abundant BC data (VIMA-80k); excessive auxiliary data can distract from BC objectives.
  • D-inBC requires an object detector (OWLv2) at inference for scenes with reference images, introducing an extra dependency and potential error source.
  • Evaluation limited to pick-and-place / rotation tasks on planar surfaces; more complex manipulation (articulated objects, contact-rich tasks) not assessed.
  • Single-image VLM backbone (LLaVA-1.5) cannot natively process temporal image sequences; history is encoded only as text, which may lose visual temporal cues.

Relevance to Vision-Language Models

LLaRA directly addresses how to efficiently transfer a pretrained VLM's conversational and spatial reasoning capabilities to a new modality (robot control) through targeted instruction-tuning data design, a broadly relevant problem in VLM adaptation. The key insight—expressing novel output modalities (actions) in the VLM's native coordinate language rather than new tokens—is transferable to other VLM grounding tasks. The self-supervised auxiliary task framework provides a recipe for domain-specific VLM enhancement without curated human annotations, complementing work on spatial VLMs (SpatialVLM) and visual grounding. For researchers tracking VLMs, this paper is a concrete case study of how data formulation choices (not model architecture changes) dominate adaptation efficiency.