Self-Improving Vision-Language-Action Models with Data Generation via Residual RL¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; self-improving VLA via residual RL, reduces reliance on human demos

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

PLD (Probe, Learn, Distill) is a three-stage post-training framework that enables vision-language-action (VLA) models to self-improve without additional human demonstrations. It trains lightweight residual RL specialists on top of a frozen VLA backbone, collects distribution-aligned trajectories via hybrid rollouts, then distills the data back into the VLA via SFT. The resulting policy achieves 99% success on LIBERO and outperforms both human teleoperation data and pure RL rollouts on generalization.

Problem¶

SFT-based post-training of VLA models requires costly human teleoperation data that is decoupled from the deployed policy's actual state distribution, leaving coverage gaps in failure and recovery states. Direct RL fine-tuning of large VLA backbones is computationally prohibitive and does not scale gracefully to multi-task settings. The central challenge is: can RL-curated data, generated with minimal human effort, match or surpass oracle human demonstrations for both in-distribution and out-of-distribution tasks?

Method¶

PLD decomposes post-training into three stages:

Stage 1 — Specialist Acquisition: The VLA backbone \(\pi_b\) is frozen. A lightweight residual Gaussian actor \(\pi_\delta(\cdot | s, a_b)\) is trained per task via off-policy actor-critic RL. The combined policy \(\bar{\pi} = \pi_b + \pi_\delta\) is optimized with Cal-QL warm-start and symmetric offline/online replay (50/50 mini-batch split). The residual magnitude is clipped to \([-\xi, \xi]\) with a curriculum scheduler to prevent early divergence from the base policy. The specialist reliably achieves \(>95\%\) success on sparse-reward tasks.

Stage 2 — Distribution-Aware Data Collection: A hybrid rollout scheme first executes the base policy \(\pi_b\) for \(t\) random steps ("base policy probing"), then hands control to the specialist \(\pi_\delta\) for the remainder. This biases the state distribution toward regions the base policy actually visits while capturing recovery behaviors from suboptimal states. The specialist is also trained on initial states sampled from \(s_0 \sim p^{\pi_b}\), aligning its competence with the generalist's trajectory distribution.

Stage 3 — Distillation via SFT: Collected multi-task hybrid trajectories are used for standard SFT on the full VLA (including the backbone). The approach is architecture-agnostic, supporting both flow-matching heads (e.g., \(\pi_0\)) and autoregressive token heads (e.g., OpenVLA).

Key Contributions¶

A plug-and-play autonomous post-training recipe (PLD) that requires no additional human demonstrations after an initial base policy SFT
Residual RL formulation for efficient specialist acquisition: freezes the expensive VLA backbone, trains only a small Gaussian residual actor, enabling multi-task RL at a fraction of the compute of direct VLA RL fine-tuning
Distribution-aware hybrid data collection via base policy probing, which explicitly captures out-of-distribution and recovery states missed by both human demos and pure RL rollouts
Architecture-agnostic SFT distillation compatible with flow-matching and autoregressive VLA action heads
Empirical demonstration that RL-generated, policy-aligned data can surpass oracle teleoperation data on generalization benchmarks

Results¶

LIBERO benchmark (\(\pi_0\)): PLD achieves 97.2% average SR vs. 93.4% baseline SFT; on LIBERO-Goal, +7.9 pp absolute gain
LIBERO benchmark (OpenVLA): 99.2% average SR vs. 91.8% baseline; LIBERO-Goal improved +15.7 pp
SimplerEnv: >50% performance gain over SFT baseline
Zero-shot generalization (LIBERO-90): PLD data trained on 10% of tasks achieves 24.4% SR on unseen tasks; self-bootstrap rollout data fails to generalize; human data achieves comparable zero-shot but lower in-distribution performance
Real-world Franka arm (cube pick-up): +DPLD achieves 30/30 vs. +DRLPD 16/30 and +DHuman 10/30
Real-world Franka arm (peg insertion): All methods 30/30
YAM bi-manual GPU insertion: Continuous 4-stage insertion/unplugging cycle for ≥1 hour without human intervention after ≤8 hours of training per subtask
RL specialist efficiency: PLD outperforms WSRL and RLPD baselines across 8 LIBERO-90 tasks within 250k interaction steps

Limitations¶

Stage 1 requires training a separate residual specialist per task, which does not scale automatically to thousands of tasks without significant compute and environment access
The approach still requires an initial SFT base policy (200 human teleoperation trajectories in real-world experiments), so it is not fully zero-human-effort
Probing horizon requires tuning; too long leads to saturation and potentially harmful distribution shift
The reward classifier for multi-stage tasks (GPU insertion) is hand-designed and task-specific
Sim-to-real gap is not fully characterized; real-world results are limited to two task families on specific hardware

Relevance to Vision-Language Models¶

PLD directly addresses a core bottleneck for VLMs extended to embodied control: the data bottleneck of SFT-only post-training. By treating the VLA's frozen vision-language backbone as a prior for RL exploration, PLD offers a scalable mechanism to generate high-quality, deployment-aligned training data without human labelers — a direct analog to RLHF/RLAIF for language VLMs. The architecture-agnostic distillation (flow-matching and autoregressive heads) and demonstrated transfer to unseen tasks position PLD as a general post-training recipe applicable to any VLA that combines a VLM backbone with an action head, including models like \(\pi_0\) and OpenVLA. For researchers tracking VLMs, this work establishes that RL-curated data can surpass human demonstrations on out-of-distribution generalization, suggesting that autonomous data flywheels may become a standard component of future VLA training pipelines.