Skip to content

Sim2Real VLA: Zero-Shot Generalization of Synthesized Skills to Realistic Manipulation

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; zero-shot sim-to-real VLA transfer via synthesized skill generalization

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Sim2Real-VLA is a Vision-Language-Action model trained exclusively on synthetic data that achieves zero-shot transfer to real-world manipulation by redesigning the model architecture rather than improving simulation fidelity. It uses a dual-system design (high-level chain-of-affordance planner + low-level affordance-executing actor) to filter manipulation-irrelevant domain features. It outperforms prior VLA baselines by over 35% on real-world tasks spanning bimanual, dexterous, and long-horizon manipulation.

Problem

VLA models trained on synthetic/simulated data suffer a severe Sim2Real gap when deployed in the real world. Prior approaches address this by building more photo-realistic simulators or world models, but residual appearance factors (lighting, background texture) remain costly to model yet have been shown to be largely irrelevant to manipulation performance. There is no principled architecture-level solution that enables zero-shot Sim2Real transfer without real-world fine-tuning.

Method

Sim2Real-VLA adopts a dual-system architecture that routes information through affordance signals, which can be reliably extracted in simulation and are inherently domain-invariant:

Planning System (Chain-of-Affordance). Given a language command \(l\) and multi-view observations, the planner predicts a sequence of affordances \([q_0, \ldots, q_K]\), where each \(q_k\) is a geometrically structured set of keypoints encoding a key end-effector pose for an atomic subtask. Object-oriented adaptation learns to recover object masks \(m^i_t\) from observations via a probabilistic predictor, trained with domain randomization (DR). DR is LLM-guided (GPT-5 ranks which features to randomize), applied at the per-timestep level rather than per-trajectory, resampling action-invariant features (lighting, texture, backgrounds) at high frequency. The planning objective minimizes: $\(\mathcal{L}(\phi) = \mathbb{E}_{\hat{\tau},[q_k]}\left[-\sum_{k,t} \log p^A_\phi(q_{k,t}|q_{k-1,t}, \hat{m}_t, o^\xi_t, \ldots, o^\xi_{t-H}, l) \cdot p^d(o^\xi_t|\hat{o},\xi)\right]\)$

Acting System (Affordance Execution). A low-level policy \(\pi_\omega(a_t,\ldots,a_{t+M}|q_k, \hat{m}_t, o_{t-H},\ldots,o_t)\) executes each affordance. Actions are represented in a tokenized space via Discrete Cosine Transform (DCT) compression of normalized action segments followed by Byte-Pair Encoding (BPE), producing compact tokens \(a^{DCT}\). For bimanual tasks, left/right arm controllers \(\pi^l_\omega\) and \(\pi^r_\omega\) are independently parameterized (arm-decoupled estimation). A validation module dynamically determines whether the current affordance has been achieved before advancing to \(q_{k+1}\).

Automatic Data Pipeline. Real2Sim projection maps teleoperation/video demonstrations into simulation; Generative Scene Scaling samples diverse configurations across scene-level and robot-level DR features; Automatic Skill Acquisition uses a VLM-equipped agent to decompose tasks, compute IK-derived trajectories, and extract affordance supervision.

Key Contributions

  • Architecture-level Sim2Real solution: closes the domain gap by filtering manipulation-irrelevant features via affordance-driven, object-oriented representations rather than building higher-fidelity simulators.
  • Chain-of-Affordance planning module that grounds embodied reasoning in geometric keypoints rather than language-only chain-of-thought.
  • Tokenized action space using DCT + BPE that compresses continuous joint-angle trajectories into compact discrete tokens.
  • Arm-decoupled estimation enabling reliable bimanual manipulation without cross-arm attention confusion.
  • LLM-guided, per-timestep domain randomization that resamples action-invariant features at higher frequency than prior trajectory-level DR.
  • Fully automated Sim2Real-VLA training pipeline (Real2Sim → Scene Scaling → Skill Acquisition) requiring no manual real-world demonstrations.

Results

  • Sim2Real-VLA achieves >35% higher task success than prior VLA baselines when both are trained on synthetic data only and tested zero-shot in real environments.
  • Evaluated on six manipulation tasks of increasing complexity: Single-Arm Water Pour, Dual-Arm Water Pour, Table Rearrangement, Items Hand-Over and Place, Basket Pick-and-Place, Pan Open-Pick-and-Place.
  • Baselines tested include OpenVLA, RTs, RDT, Ď€0, and other dual-system VLA frameworks trained under the same synthetic-only regime.
  • Zero-shot Sim2Real capability generalizes reliably across diverse domain shifts (varying lighting, textures, backgrounds, object poses), verified quantitatively.
  • Attention map visualizations confirm the model focuses on manipulation-relevant regions during real-world deployment.

(Note: per-task numeric success rates are not reported in the provided text excerpt.)

Limitations

  • Performance depends entirely on simulation data quality and coverage; tasks without faithful Real2Sim projection may be underrepresented.
  • Affordance extraction in simulation requires access to full object spatial/physical/semantic properties; this privileged information is unavailable at real-world inference time, creating a training-inference asymmetry.
  • Arm-decoupled estimation assumes the two arms operate sufficiently independently; tightly coupled bimanual tasks with shared affordances may degrade.
  • DCT + BPE tokenization introduces quantization error; behavior at the resolution boundary (fine-grained dexterous motion) is not analyzed.
  • LLM-guided DR (using GPT-5) adds external API dependency and cost to the data generation pipeline.
  • Long-horizon task evaluation is limited to the six tasks described; generalization to unseen task structures beyond those categories is unverified.

Relevance to Vision-Language Models

Sim2Real-VLA is directly relevant to VLM-grounded robotics, demonstrating that VLM backbone models can be structured to suppress domain-specific visual noise and focus on task-critical geometric representations. The affordance-driven planning layer acts as a structured intermediate representation that decouples high-level semantic reasoning (handled by the VLM) from low-level motor execution, a design pattern increasingly adopted across frontier VLA systems (π0, Helix, Gr00N). The LLM-guided domain randomization strategy shows how large language/vision models can play an active role in automated training data curation, not just policy inference. For VLM researchers, this work provides empirical evidence that architecture choices—specifically object-oriented observation adaptation and tokenized action spaces—matter more for real-world generalization than visual realism of training data.