Semantic Temporal Abstraction via Vision-Language Model Guidance for Efficient Reinforcement Learning

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VanTA uses pretrained VLMs to semantically guide vector-quantized skill discovery for offline hierarchical RL. By iteratively relabeling VQ-derived trajectory segments with VLM-assigned indices and updating the codebook via exponential moving average, the method extracts interpretable, temporally coherent primitive skills without manual annotation. This significantly improves long-horizon offline RL across manipulation, navigation, and open-world environments.

Problem

Existing temporal abstraction methods for offline RL either discover skills unsupervisedly (yielding semantically vacuous, fragmented segments that corrupt hierarchical policy learning) or require heavy human annotation. Neither scales: unsupervised VQ produces scattered, context-free codebooks; heuristic fixed-horizon segmentation produces meaningless divisions; human-labeled methods contradict automation goals.

Method

VanTA operates in two stages:

Stage 1 – Semantic Skill Extraction. A modified VQ-VAE encoder maps observations to continuous latent codes, then quantizes them to the nearest codebook entry. The policy decoder replaces the reconstruction decoder (predicting actions rather than states). Every I epochs, the initial and terminal observations of each VQ-derived segment are passed to a pretrained VLM, which returns a codebook index j = VLM(s̄). The codebook is then updated via exponential moving average toward the encoder embeddings assigned to index j by the VLM: z_j ← z_j · λ + Σ h_j · (1 − λ) A smoothness loss (L1 on consecutive latent differences) encourages temporal coherence, and a "no suitable skill" fallback handles VLM hallucinations. This alternation repeats until convergence, progressively making segments semantically interpretable.

Stage 2 – Policy Extraction. A high-level policy selects discrete skills from the compact codebook using conservative offline DQN (Bellman backup over skill-level transitions). A low-level policy is trained with importance-weighted behavior cloning conditioned on both state and the assigned skill token (IQL weighting: exp(Q(s,a) − V(s))).

Key Contributions

• VLM-guided codebook refinement: iterative relabeling loop that grounds VQ latent space in VLM semantic knowledge without human annotation
• Modified VQ-VAE architecture: replaces reconstruction decoder with a policy decoder, targeting behavior abstraction rather than image reconstruction
• Theoretical analysis: proves that VLM-induced higher autocorrelation in skill action sequences shrinks the effective policy space (|Πα| ≪ |Π_ori|), reducing suboptimality bounds compared to flat offline RL
• Empirical validation on three diverse long-horizon offline RL benchmarks (Franka Kitchen, MiniGrid, Crafter) with both proprioceptive and visual observations
• Low-data regime benefit: VLM knowledge supplements scarce offline data, with 28.6% improvement on kitchen-complete with only 10% data

Results

• Franka Kitchen (vs. best baseline LDCQ): VanTA scores 69.2 / 71.2 / 68.5 on complete/partial/mixed vs. LDCQ's 52.8 / 67.8 / 62.3
• MiniGrid (8 tasks): VanTA matches or exceeds all baselines; largest gain on KeyCorridorS3R3-mixed (0.61 vs. 0.51 for IQL)
• Crafter-partial: VanTA 5.46 vs. IQL 2.75, LDCQ 0.96, BC 2.69
• Low-data (10% Kitchen-complete): VanTA 17.1 vs. IQL 13.3, BC 7.6, CQL 2.7
• Temporal correlation / policy space reduction: VanTA achieves r₁ = 0.69/0.59/0.68 on Kitchen tasks (vs. 0.42/0.44/0.67 without VLM), reducing policy space to 41–69% of original vs. 53–80% without VLM guidance

Limitations

• Proper VQ initialization is essential; without it, VLM query complexity explodes since single observations lack sufficient context for meaningful VLM responses
• Mode collapse in VQ (a common unsupervised learning failure) cannot be corrected by VLM relabeling — the VLM cannot fix degenerate codebooks post-collapse
• VLM outputs are stochastic and can hallucinate; exception handling (the "no suitable skill" option) mitigates but does not eliminate noise
• Query complexity to the VLM, though reduced by initialization, remains a practical bottleneck for large offline datasets
• Skill length K is assumed uniform in theory, which may not hold in practice

Relevance to Vision-Language Models

VanTA demonstrates a novel use pattern for VLMs: as a discriminative semantic oracle that labels latent clusters rather than as a generative planner, exploiting VLMs' classification reliability over their generation reliability. This is directly relevant to researchers studying how frozen VLMs can be injected into non-language pipelines (RL, representation learning) without fine-tuning. The work extends the line of VLM-grounded embodied AI (SayCan, LDCQ) by operating offline and on the latent codebook level rather than on raw action planning, suggesting a new modality for VLM knowledge distillation into sequential decision-making systems.