EVLP: Learning Unified Embodied Vision-Language Planner with Reinforced Supervised Fine-Tuning¶
🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link
Why this paper was selected
ICLR 2026; reinforced supervised fine-tuning for unified embodied VL planner
Ask a follow-up
Open an assistant pre-loaded with this paper's context.
💬 Ask ChatGPT✦ Ask Claude
TL;DR¶
EVLP is a unified multimodal framework that jointly generates stepwise language actions and visual subgoal images for long-horizon robotic manipulation. It combines a dual-tower vision module, bidirectional dynamics pretraining, and Reinforced Supervised Fine-Tuning (RSFT) to enforce spatial consistency between generated text plans and predicted next-state images. On LoHoRavens, EVLP outperforms the strongest baseline (PERIA) by ~10–15 percentage points in task success rate across all evaluated tasks.
Problem¶
Embodied long-horizon planning requires simultaneous textual reasoning (what to do) and visual imagination (how the scene will look after acting), but existing methods treat these as separate pipelines, causing semantic inconsistency between language plans and visual subgoals. Unified multimodal generative models have not been adapted to handle three embodied-specific challenges: (1) spatial detail understanding beyond high-level semantics, (2) temporal state-transition reasoning, and (3) training objectives that prioritize task-relevant dynamics over pixel-level visual fidelity.
Method¶
Unified Architecture. A dual-tower vision module handles understanding and generation separately: SigLIP extracts high-level semantic features (weights frozen), while a trainable low-level spatial encoder captures positional detail via reconstruction loss. Both are fused through an adapter into a pretrained LLM. For generation, Open-MAGVIT2 provides a lookup-free quantizer (codebook size \(K = 262{,}144\)) that encodes \(256 \times 256\) images into \(16 \times 16\) discrete tokens. Instead of autoregressive token-by-token decoding, EVLP introduces learnable image tokens input to the transformer and predicts all discrete token IDs in one forward pass, directly modeling \(x_{0:N} \sim p(\cdot \mid c)\).
Dynamic Perception Pretraining. On transition tuples \(T = \{x_t, a_t, x_{t+1}\}\), two tasks are co-trained: - Inverse Dynamic Task: predict action \(a_t\) from \((x_t, x_{t+1})\), maximizing \(\mathcal{L}_{\text{Inverse}} = -\mathbb{E}\left[\frac{1}{L}\sum_{i=1}^L \log P(a_t^{(i)} \mid a_t^{(<i)}, x_t, x_{t+1}; \theta)\right]\) - Forward Dynamic Task: predict \(x_{t+1}\) from \((x_t, a_t)\), maximizing \(\mathcal{L}_{\text{Forward}} = -\mathbb{E}\left[\log P(x_{t+1}^{(0:N)} \mid x_t, a_t; \theta)\right]\)
Reinforced Supervised Fine-Tuning (RSFT). SFT jointly maximizes the log-likelihood of action tokens and next-state image tokens given a high-level instruction \(g\) and current observation \(x_t\): $\(\mathcal{L}_{\text{SFT}} = -\mathbb{E}\left[\frac{1}{L}\sum_{i=1}^L \log P(a_t^{(i)} \mid a_t^{(<i)}, g, x_t) + \log P(x_{t+1}^{(0:N)} \mid g, x_t, a_t^{0:L})\right]\)$ A policy gradient term reinforces dynamic consistency: \(K\) samples \(x_k \sim P(x_{t+1} \mid g, x_t, a_t; \theta)\) are drawn in a single forward pass, scored by a Dynamic Alignment reward \(R(x)\), batch-normalized into advantages \(A_k\), and optimized via: $\(\mathcal{L}_{\text{RL}} = -\mathbb{E}\left[\frac{1}{K}\sum_{k=1}^K A_k \cdot \log P(x_{t+1}^k \mid g, x_t, a_t; \theta)\right]\)$ The final objective is \(\mathcal{L} = \mathcal{L}_{\text{SFT}} + \lambda \cdot \mathcal{L}_{\text{RL}}\), with MLE stabilizing training against RL's high-variance gradients.
Key Contributions¶
- One-step non-autoregressive image generation integrated into an LLM backbone via learnable image tokens, enabling \(K\) independent samples per forward pass without multi-step overhead.
- Dual-tower vision module (SigLIP for semantics + spatial encoder for detail) that explicitly compensates for SigLIP's spatial blind spots.
- Bidirectional dynamics pretraining combining inverse and forward dynamic tasks to ground the model in physical state transitions before task-specific fine-tuning.
- RSFT training paradigm that combines MLE (distribution alignment) with policy gradient (per-sample dynamic preference) without extra sampling cost at training time.
Results¶
LoHoRavens benchmark (success rate, mean ± std over 5 seeds):
| Task | PERIA (best baseline) | EVLP |
|---|---|---|
| Blocks Stacking | 63.9 ± 5.8 | 79.4 ± 7.9 |
| Blocks Sort | 65.0 ± 6.4 | 77.3 ± 4.3 |
| Letters Matching | 72.3 ± 7.1 | 82.5 ± 6.1 |
| Letters Shape | 60.6 ± 5.2 | 75.3 ± 4.4 |
| Letters Orders | 65.2 ± 6.7 | 78.2 ± 7.3 |
| Letters Spell | 71.1 ± 7.5 | 81.8 ± 6.5 |
Meeting Preparation ablations (Table 2): - Full EVLP: SR=67.6, LA=87.0, LPIPS=0.051, SSIM=0.95 - Removing forward dynamics (w/o FDM): SR drops to 26.8, LPIPS=0.192 — largest single-component degradation - Removing RL: SR=62.2 (−5.4 pp), confirming RSFT contribution - RL only (no SFT): SR=0.0, confirming SFT is essential for stability
Limitations¶
- Evaluated exclusively in simulation (Ravens-based LoHoRavens, in-house Meeting Preparation); no real-robot transfer demonstrated.
- The Dynamic Alignment reward function \(R(x)\) is defined in Appendix C.1 (not provided in the main text); its generalizability to novel task domains is uncharacterized.
- One-step generation trades per-image diversity for efficiency; the fixed codebook size (\(K=262{,}144\)) may limit reconstruction fidelity for high-complexity scenes.
- The low-level policy executing the generated plans is treated as a black box; planning errors and execution errors are conflated in the success rate metric.
Relevance to Vision-Language Models¶
EVLP extends the unified text-image generation paradigm (e.g., LlamaGen, Janus) into the embodied domain, demonstrating that a single LLM backbone can jointly model language plans and visual subgoals without separate diffusion rendering. The RSFT approach is directly applicable to any VLM that supports one-step discrete image generation — it provides a principled way to inject physically grounded rewards into MLE fine-tuning without the multi-step sampling overhead that makes RL expensive for autoregressive models. The dual-tower perception design (semantic + spatial encoders) addresses a documented limitation of SigLIP-based VLMs in spatial understanding, which is broadly relevant to grounded VLM research. This work positions embodied manipulation as a forcing function for VLM capabilities that static image-text benchmarks cannot expose.