Skip to content

Exploring the Limits of Vision-Language-Action Manipulation in Cross-task Generalization

๐Ÿ•’ Published (v1): 2025-01-01 ยท Source: NeurIPS ยท Venue: NeurIPS 2025 ยท link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

๐Ÿ’ฌ Ask ChatGPTโœฆ Ask Claude

TL;DR

Current VLA models fail to generalize zero-shot to unseen manipulation tasks despite large-scale pretraining. This paper introduces AGNOSTOS, a simulation benchmark with 23 unseen tasks at two difficulty levels, and X-ICM, a cross-task in-context manipulation method that uses dynamics-guided demonstration retrieval to prompt LLMs for action prediction on unseen tasks. X-ICM (72B) achieves 30.1% average success rate, outperforming the best prior VLA model (ฯ€0 at 17.5%) by 12.6 points absolute.

Problem

Existing VLA benchmarks (Colosseum, GemBench, CALVIN, etc.) evaluate within-task visual generalization โ€” novel scenes or object attributes within known tasks โ€” but not zero-shot cross-task generalization, where the task itself (object category + motion primitive) has never been seen during training. No reproducible, systematic benchmark existed for this harder evaluation regime, leaving the cross-task generalization limits of VLA models unmeasured.

Method

AGNOSTOS benchmark: Built on RLBench with 18 seen training tasks (3,600 demos, 200/task). Test set: 23 held-out unseen tasks split into Level-1 (13 tasks sharing partial semantics โ€” similar objects or motions with seen tasks) and Level-2 (10 tasks with no overlap in object categories or motion types). Three evaluations per task seed, 25 rollouts each.

X-ICM (Cross-task In-context Manipulation): 1. Dynamics-guided sample selection: A diffusion model (initialized from InstructPix2Pix) is trained on seen demonstrations to predict final visual observations from initial observations + language descriptions. The frozen model extracts 2ร—1024-dim dynamic features (visual + language latents). For an unseen task, cosine similarity between its dynamic feature and all seen demonstration features identifies the top-K=18 most relevant demonstrations. 2. Cross-task in-context prediction: Selected demonstrations are textualized: object 3D positions detected via GroundingDINO and projected to robot-base coordinates (normalized to 100ยณ grid), key-action sequences (7-DoF end-effector states at gripper-state-change or near-zero-velocity keyframes, angles discretized to 72 bins). These are formatted as [task_instruction, objects] โ†’ [key_actions] pairs and concatenated as a prompt to an off-the-shelf LLM (Qwen2.5-Instruct 7B or 72B) to predict the action sequence for the unseen task.

Key Contributions

  • AGNOSTOS: first benchmark to evaluate zero-shot cross-task generalization for VLA models, with two calibrated difficulty levels and broad model coverage (foundation, human-video, and in-domain VLA families).
  • X-ICM: first extension of in-context learning for robotic manipulation to the cross-task zero-shot regime.
  • Dynamics-guided sample selection: diffusion-based dynamic representation learning to retrieve cross-task-relevant demonstrations without requiring task-label alignment.
  • Comprehensive empirical evaluation of 14+ VLA models revealing consistent near-zero cross-task performance across all prior approaches.

Results

  • X-ICM (72B): 37.6% Level-1 / 20.3% Level-2 / 30.1% overall โ€” best overall model.
  • X-ICM (7B): 28.6% Level-1 / 16.9% Level-2 / 23.5% overall.
  • Best prior VLA model: ฯ€0 at 21.7% Level-1 / 12.0% Level-2 / 17.5% overall.
  • X-ICM (72B) vs. ฯ€0: +15.9% Level-1, +8.3% Level-2, +12.6% overall.
  • X-ICM (7B) vs. VoxPoser: +7.9% overall (paper abstract); vs. ฯ€0: +6.0% overall.
  • Dynamics-guided selection ablation: removing it drops X-ICM (72B) from 30.1% to 25.2% (โˆ’4.9%) with much higher variance (std 3.2 vs. 1.0).
  • Optimal in-context demo count: K=18; performance plateaus after K=12.
  • LLM backbone matters: Qwen2.5-7B (23.5%) >> Llama3.0-8B (15.1%) >> Deepseek-R1-Distill-7B (9.5%) at same scale.
  • X-ICM (72B) fails on 0 of 23 tasks; X-ICM (7B) fails on 2; all prior models fail on โ‰ฅ8.
  • Real-world (xArm7, 5 tasks, 20 rollouts each): put block in bin 70%, stack blocks 50%, put bottle in box 30%, stack cups 30%, push button 10%.
  • Long-horizon real-world (3 sequential subtasks, 20 rollouts): 40% / 25% / 5% cumulative โ†’ 5% end-to-end.

Limitations

  • Level-2 (fully novel object+motion) performance remains low even for X-ICM (72B) at 20.3%, indicating LLMs cannot reliably extrapolate beyond in-context demonstrations when both object and motion are unseen.
  • Visual information is used only via textualized object positions; raw visual context (e.g., texture, shape, spatial layout beyond keypoints) is discarded, losing potentially relevant grounding.
  • Long-horizon cross-task generalization degrades sharply due to error accumulation; 5% end-to-end on a 3-subtask sequence.
  • Uses ground-truth object positions in simulation for fair comparison with zero-shot baselines โ€” relaxing this to estimated positions may hurt performance.
  • Evaluation limited to RLBench simulator; generalization to other simulators or diverse real-world embodiments untested at scale.
  • Dynamics diffusion model trained only on seen tasks; its feature space may not transfer well to highly novel Level-2 dynamics.

Relevance to Vision-Language Models

This paper directly stress-tests whether large-scale VLA pretraining (on cross-embodiment robot data or human video) confers compositional cross-task generalization โ€” a core hypothesis in the VLM-for-robotics line of work โ€” and finds it largely does not. The X-ICM approach reframes the problem as an in-context learning task over an LLM backbone, exploiting the cross-task semantic knowledge already encoded in pretrained language models rather than in task-specific robot policies, offering a complementary paradigm to end-to-end VLA fine-tuning. The dynamics diffusion module is a practical technique for grounding retrieval in task-relevant visual change rather than superficial language similarity, a challenge shared broadly across multimodal retrieval and tool-use in VLMs. AGNOSTOS fills a critical evaluation gap and provides a reproducible testbed for measuring genuine compositional generalization, which is increasingly relevant as the community moves toward open-world deployment of VLMs in embodied settings.