Test-Time Perturbation Learning with Delayed Feedback for Vision-Language-Action Models¶

🕒 Published (v1): 2026-04-20 11:25 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

Test-time perturbation learning corrects VLA trajectory overfitting; CVPR 2026

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Vision-Language-Action (VLA) models suffer from trajectory overfitting—memorizing spurious correlations between visual context and action sequences rather than grounding decisions in task semantics. This paper proposes PDF (Perturbation learning with Delayed Feedback), a verifier-free test-time adaptation framework that combines uncertainty-guided data augmentation with a lightweight logit-perturbation module trained on delayed episodic feedback. PDF achieves +7.4% success rate on LIBERO and +10.3 HNS on Atari-57 without modifying base model parameters.

Problem¶

VLAs trained via imitation learning overfit to trajectory-specific visual patterns (gripper appearance, background textures, object poses), reproducing memorized action traces even when the target object is occluded or absent. Existing test-time adaptation (TTA) methods are either verifier-based (high compute, poor generalization) or verifier-free but rely on entropy minimization, which amplifies errors when the model is miscalibrated and overconfident in wrong actions.

Method¶

PDF operates entirely at test time and leaves base model weights frozen. It has two components:

1. Uncertainty-Based Action Voting. At each step $t$, action-logit uncertainty is measured as normalized Shannon entropy: $$U_t = -\frac{1}{\log K} \sum_{k=1}^{K} p(a_k | s_k) \log p(a_k | s_k)$$ An adaptive augmentation budget $N_t = N_{\max} \cdot U_t$ controls how many augmented views of the pixel observation are generated. All views (original + augmented) are encoded into multimodal features $f_t$, which feed both a frozen LM head $h_\varphi(\cdot)$ and a trainable Perturbation head (P head) $h_\theta(\cdot)$: $$\tilde{z}_t = h_\varphi(f_t) + \lambda h_\theta(f_t)$$ The final action is selected by dimension-wise majority voting over candidate actions from all views.

2. Delayed Feedback-Guided Adaptation. After each episode ends, features and voted logits stored in a rollout buffer $\mathcal{D}$ are used to update only $h_\theta$ via: $$\mathcal{L}_\text{PDF} = -(r - b)\log \pi_\varphi + \lambda_\text{KL}\, \mathbf{1}[r > b]\, \text{KL}(\pi_\varphi \| \tilde{\pi})$$ The first term is a REINFORCE-style objective; the KL regularizer (gated on positive feedback) constrains the perturbed policy toward the base policy, preventing overconfident drift. The P head (9M parameters) is the only component updated.

Key Contributions¶

Formal identification and analysis of trajectory overfitting in VLAs, demonstrated via attention maps and masking ablations.
PDF: a plug-and-play, verifier-free TTA framework that requires no base-model gradients and only 9M trainable parameters (vs. 93–130M for baselines).
Adaptive augmentation budget scheduler that scales augmentation to decision uncertainty, avoiding noise accumulation at low-uncertainty steps.
Dimension-wise majority voting over augmented views, which is more robust than action-wise voting for multi-dimensional continuous action spaces.
Empirical validation across both robotic manipulation (LIBERO, 4 suites) and visual game control (Atari-57).

Results¶

LIBERO (OpenVLA base): PDF achieves 0.77 average success rate (SR), best mean rank 2.5 across 4 suites; +7.4% over vanilla OpenVLA (0.69 reproduced SR). Largest gain on LIBERO-Long (+4.1 over best baseline, SR = 0.59); LIBERO-Spatial SR = 0.90, LIBERO-Goal SR = 0.86.
Atari-57 (Jat base): PDF achieves 1.07 HNS vs. 0.97 baseline (+0.10, reported as +10.3 human-normalized score); 47/57 games improve; largest gains: BOXING +0.60, TIME PILOT +0.53, ATLANTIS +0.49.
vs. MG-Select (closest verifier-free TTA baseline): PDF improves average LIBERO SR by ~6 points (0.77 vs. 0.71).
Ablation: removing delayed feedback (DF) causes the largest drops (Object SR falls from 0.72 to 0.50); removing data augmentation (DA) causes smaller but consistent degradation; both components are complementary.
Loss ablation: full loss outperforms w/o KL and w/o REINFORCE variants; KL contributes more on Atari, REINFORCE-style term contributes more on object-manipulation tasks.
Augmentation budget: $N_{\max} = 3$ is optimal; larger budgets degrade performance due to noise accumulation.

Limitations¶

Delayed feedback assumes episodic binary or scalar rewards; inapplicable to settings without any environment feedback signal at episode boundaries.
Adaptive augmentation still introduces computational overhead proportional to uncertainty; real-time latency on resource-constrained hardware is not characterized.
Degradation observed on 10/57 Atari games (max: BATTLE ZONE −10.7% HNS), suggesting the method can harm performance in specific game dynamics.
Evaluation is limited to simulation; real-robot transfer with noisy or sparse feedback is not demonstrated.
Sensitivity to $N_{\max}$ is benchmark-dependent, requiring per-task tuning for optimal results.

Relevance to Vision-Language Models¶

PDF directly addresses a failure mode of VLMs deployed as action generators (VLAs): the visual attention mechanism latches onto spurious correlations rather than semantically relevant objects, an issue that manifests in grounded reasoning tasks more broadly. The delayed-feedback perturbation approach—training only a small adapter head at inference time using environmental signals—is a practical alternative to full RLHF or fine-tuning for grounding correction in VLMs. The uncertainty-driven augmentation strategy connects to test-time compute scaling research in LLMs, showing that adaptive sampling budgets can improve multimodal decision quality without architectural changes. For VLM practitioners, the trajectory overfitting diagnosis (attention map analysis + masking experiments) offers a diagnostic methodology transferable to any VLM deployed in an interactive loop.