Foresight Residual RL for Long-Horizon Robot Manipulation with Vision-Language-Action Models¶

🕒 Published (v1): 2026-07-17 21:00 UTC · Source: Arxiv · link

Why this paper was selected

VLA + residual RL for long-horizon manipulation; Boularias lab; addresses credit assignment gap

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Vision-Language-Action (VLA) policies fine-tuned with residual RL on individual subtasks fail in chained long-horizon assembly because all geometrically successful terminal states are treated equally, ignoring how handoff quality affects downstream success. Foresight Residual RL augments each subtask's sparse reward with an offline-estimated visual foresight value—the predicted probability that the next subtask will succeed given the current terminal state—trained via backward induction. On a three-phase wrench-nut tightening task, the method raises full-task success from 54.5% to 85.6% without degrading per-subtask success rates.

Problem¶

Standard residual RL over frozen VLA base policies optimizes each subtask independently with a constant sparse success reward ($r_t = \beta_k(s_{t+1})$). This treats all geometrically successful terminal states as equally valuable, ignoring causal coupling between subtasks: a kinematically valid but shallow wrench insertion will fail under rotation forces. Distributional-overlap methods (initiation overlapping) address terminal-state mismatch via a proxy constraint rather than directly maximizing downstream success probability, and existing hierarchical methods (MAXQ, options) require simultaneous training of all levels, making them incompatible with frozen pre-trained VLA bases.

Method¶

Foresight value function. For phase $k < K$ with fixed downstream policies $\pi_{k+1:K}$, the foresight value is: $$V_{k+1:K}(s) = \mathbb{E}_{\pi_{k+1:K}}\!\left[\sum_{t=0}^{T_{k+1:K}} \gamma^t \beta_K(s_{t+1})\,\middle|\, s_0^{(k+1)} = s\right]$$ approximated in practice by the immediate next-subtask success probability $V_{k+1}(s) = \Pr[\text{subtask } k{+}1 \text{ succeeds} \mid s_0^{(k+1)}=s,\,\pi_{k+1}]$.

Offline visual foresight predictor (Algorithm 2). (1) Roll out frozen base VLA policy $\pi_k^0$ to collect $N=2{,}560$ successful terminal states. (2) From each terminal state, run $K_\text{rep}=5$ independent rollouts of the downstream policy; record empirical success count $\kappa_i$. (3) Train a neural predictor $p_\phi: \mathcal{O} \to [0,1]$ with binomial negative log-likelihood: $$\mathcal{L}(\phi) = -\sum_{i=1}^N \left[\kappa_i \log p_\phi(o_i) + (K_\text{rep}-\kappa_i)\log(1-p_\phi(o_i))\right]$$ Architecture: DINOv2-S/14 CLS tokens from two camera views fused via self-attention + linear layer + sigmoid head, with LoRA rank-8 fine-tuning.

Foresight-modulated reward. Training reward for phase $k$ becomes: $$r_t = \beta_k(s_{t+1}) \cdot p_\phi(o_{t+1})$$ This multiplies the sparse success signal by predicted downstream success probability, steering the residual policy toward terminal states with high handoff quality.

Backward foresight induction (Algorithm 1). Phases are trained in reverse order $k = K, K{-}1, \ldots, 1$. The last phase trains with standard reward; each earlier phase trains with the foresight-modulated reward using the already-fixed downstream policies. Residual actions are $a_t = \pi_k^0(o_t) + \alpha \cdot \pi_k^\text{res}(o_t)$ with $\alpha=0.1$. Residual policy: frozen DINOv2-S/14 wrist-camera features → CNN → 2-layer LSTM (256) → MLP → actor/critic; trained with PPO across 128 parallel environments in Isaac Gym.

Key Contributions¶

Foresight Residual RL framework: formally defines the foresight value function as the MAXQ completion function restricted to a linear task graph, and shows that multiplying the local terminal reward by this value aligns subtask optimization with global success.
Offline predictor pipeline: amortizes expensive Monte Carlo downstream evaluation into a one-time offline data collection step, producing a calibrated image-based foresight predictor that provides instantaneous, low-variance reward signals during RL training.
Backward foresight induction: a single-pass phase-reversed training schedule that decouples foresight estimation from subtask policy training, making the approach compatible with frozen pre-trained VLA base policies ($\pi_0$).
Empirical validation: demonstrates on a contact-rich 3-phase wrench assembly task that foresight improves full-task success by +31.1 percentage points over standard residual RL while leaving per-subtask success rates unchanged.

Results¶

Full-task success: Foresight Residual RL 85.6% vs. standard subtask residual RL 54.5% (+31.1 pp) on the 3-phase wrench-nut tightening task in Isaac Gym.
Per-subtask success rates are statistically unchanged between with/without foresight, confirming the gain is entirely attributable to improved terminal-state quality, not subtask capability.
Foresight predictor accuracy (held-out test, threshold 0.5):
Grasp→Insert: 86.2% vs. 73.9% majority-class baseline.
Insert→Rotate: 90.4% vs. 93.9% majority-class baseline (smaller margin; high-success regime).
Predictor training: <5 minutes with $N=2{,}560$ terminal states and $K_\text{rep}=5$ rollouts per boundary.

Limitations¶

Evaluated on a single simulated task (wrench-nut assembly in Isaac Gym); no real-robot or transfer experiments reported.
One-step foresight approximation ($V_{k+1}$ instead of $V_{k+1:K}$) is exact only for the penultimate phase or when later phases have constant conditional success rates; deeper chaining may require multi-step rollouts.
Predictor distribution shift: as the residual policy improves, it may visit terminal states outside the base policy's distribution used for predictor training; small $\alpha=0.1$ is relied upon to bound this shift without re-training the predictor.
$K_\text{rep}=5$ rollouts per terminal state is a coarse success-rate estimate; binomial variance is high for small $K_\text{rep}$.
Base VLA demonstrations are homogeneous (fixed expert strategy, narrow terminal-state distribution), which may limit generalization to diverse demonstration sources.
No ablation over the one-step vs. full multi-step foresight approximation, or over $N$ and $K_\text{rep}$ hyperparameters.

Relevance to Vision-Language Models¶

This work directly addresses a practical failure mode of VLA models (such as $\pi_0$) in multi-step manipulation: strong per-skill performance that collapses when skills are sequenced, a gap not closed by standard fine-tuning or residual RL. The foresight predictor is a lightweight vision module (DINOv2 + LoRA) trained on the VLA's own rollouts, making it directly composable with any frozen VLA backbone without re-training the base model. For researchers tracking VLMs in embodied settings, this offers a principled recipe for injecting cross-phase supervision into VLA residual RL that respects the frozen-model constraint common to large generalist policies. The offline estimation paradigm—distilling Monte Carlo rollout statistics into an image-based calibrated predictor—is broadly applicable to other VLA architectures and multi-phase task structures.