Skip to content

Human-assisted Robotic Policy Refinement via Action Preference Optimization

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Action Preference Optimization (APO) refines Vision-Language-Action (VLA) models post-deployment by learning from human-intervention trajectories collected during robot operation. It adapts KTO-style preference alignment (binary desirability signals) with an adaptive reweighting scheme that corrects for the mismatch between discrete action tokens and continuous action regression error. This enables stable, iterative improvement of large-scale VLA models without the instability of RL-based methods.

Problem

Pretrained VLA models rely entirely on offline expert demonstrations and cannot learn from failures encountered during deployment. Interactive imitation learning via behavior cloning fails to exploit failure trajectories, while reinforcement learning methods are too unstable for large-scale VLA fine-tuning. Additionally, adapting LLM preference optimization (e.g., DPO/KTO) directly to VLA models faces two compounding challenges: (1) robotic interactions are irreversible, making paired positive/negative samples under identical conditions infeasible; (2) autoregressive VLAs discretize continuous actions into tokens, creating a mismatch between token classification probabilities and continuous action regression loss.

Method

APO operates as a deployment-optimization loop with two components:

Human-robot collaboration framework: A base policy (OpenVLA fine-tuned via behavior cloning on expert demonstrations) is deployed while a human operator monitors and intervenes via SpaceMouse. Intervention actions are labeled desirable (c=2); the K=10 preceding policy actions are retrospectively labeled undesirable (c=0). Balanced sampling enforces 50% expert / 25% intervention / 25% failure per batch.

Action preference optimization: APO adapts KTO (prospect theory) with binary desirability signals rather than paired preferences, relaxing the irreversibility constraint. The reward is estimated as r_θ(o, â) = log π_θ(â|o) / π_ref(â|o), where π_ref is the model at the start of each loop iteration. KL divergence from π_ref serves as the penalty anchor z_0.

Adaptive reweighting: To address token-action mismatch, per-sample L1 continuous action error is computed, batch-normalized to weights w_i, then used to modulate λ_D and λ_U (importance sampling coefficients): λ_D = 1 − e^{−β_D·w}, λ_U = e^{−β_U·w}. For desirable samples, higher weight goes to high-regression-error steps; for undesirable samples, higher weight goes to steps whose continuous actions are closest to failure actions.

Key Contributions

  • A human-in-the-loop deployment framework that automatically labels failure precursors (K steps before intervention) without requiring paired trajectories.
  • A KTO-based preference alignment objective adapted for irreversible robotic interaction via binary desirability signals and KL-divergence anchoring.
  • An adaptive reweighting algorithm that bridges discrete token probabilities and continuous action regression loss via sample-wise L1-error-derived weights.
  • Demonstrated iterative (lifelong) improvement: success rate increases and human intervention frequency decreases across deployment rounds.
  • Validation across two VLA architectures (OpenVLA, Ï€0-FAST) in simulation (RoboMimic) and real-world fine-grained insertion tasks.

Results

In-distribution comparison (RoboMimic, 4 tasks, mean success rate): - Base policy: 40.5%; Dagger: 39.0%; Sirius: 39.5%; DPO: 37.0%; TPO: 41.5%; KTO: 43.5% - APO: 48.0% — consistent improvement across all tasks vs. all baselines.

Disruption generalization (mean over position/background/texture disruptions): - Best prior method (KTO): 24.0% - APO: 28.0% (disruption scenarios); APO also achieves 45.3% mean on original tasks after disruption fine-tuning vs. KTO's 39.3%.

Ï€0-FAST generalization (Coffee_D0 / StackThree_D0): - Base: 68%/64%; Dagger: 64%/66%; TPO: 48%/52% - APO: 76%/74%

Real-world insertion task (in-distribution / pos / bg / texture disruption): - Base: 65%/25%/10%/25%; Dagger: 65%/10%/10%/25%; TPO: 75%/40%/20%/45% - APO: 85%/55%/30%/55%

Lifelong learning: APO success rate rises monotonically across interaction rounds while human intervention frequency decreases; BC baseline plateaus.

Limitations

  • Experiments exclusively use autoregressive VLA models (OpenVLA, Ï€0-FAST); applicability to regression-based or diffusion-policy VLAs is untested.
  • Requires a human operator available for real-time intervention during deployment, limiting fully autonomous refinement.
  • K=10 pre-intervention labeling as "undesirable" is a heuristic; optimal K is task-dependent and not analyzed.
  • Disruption generalization gains are modest (APO 28.0% vs. KTO 24.0% mean), and texture disruption remains challenging (12% for APO vs. 6% for KTO — both very low).
  • Evaluated on a single simulation suite (RoboMimic) and limited real-world tasks; broader task diversity is unvalidated.

Relevance to Vision-Language Models

APO directly addresses the post-deployment adaptation gap in VLA models, which are the robotics instantiation of VLMs extended with action heads. It introduces a principled mechanism to transfer preference alignment techniques (DPO/KTO) from language models to the continuous-action, irreversible-interaction regime of robotics, making it a practical bridge between LLM alignment research and embodied AI. The adaptive reweighting insight — using continuous regression error to reweight discrete token optimization — is a general solution to the token-action mismatch problem inherent to any autoregressive VLA. For researchers tracking VLMs, this work exemplifies how RLHF-derived alignment methodology must be rethought when the output space is physical rather than linguistic.