GoalLadder: Incremental Goal Discovery with Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GoalLadder trains RL agents in visual environments using only a single natural language instruction by incrementally discovering and ranking candidate goal states via a VLM, without requiring manual reward engineering. It achieves ~95% mean success rate across classic control and robotic manipulation tasks, nearly matching oracle agents with ground-truth rewards. The key insight is decoupling VLM-based goal discovery (sparse, pairwise queries) from reward computation (dense, unsupervised embedding distances), making the system both noise-robust and query-efficient.

Problem

Existing VLM-based reward methods for RL in visual environments suffer from two compounding issues: (a) noisy reward signals due to VLM hallucinations and domain mismatch (embedding-based methods like CLIP) or erroneous preference labels (RL-VLM-F), and (b) high VLM query costs because reward must be computed for every observed state. Methods relying on non-visual state representations (code-based LLM reward generators) fail to generalize to arbitrary visual environments. No prior method simultaneously handles noise robustness and query efficiency in pixel-based RL.

Method

GoalLadder maintains a bounded buffer of candidate goal states (|B_g| = 10) and operates in four repeating stages:

1. Collection: SAC agent collects a trajectory under the current policy.
2. Discovery: M uniformly sampled trajectory observations are each compared pairwise against the current top-rated candidate via a VLM prompt (Gemini 2.0 Flash) that asks which image is closer to the language-specified goal; observations deemed better are inserted into the buffer with a default ELO rating.
3. Ranking: M pairs of existing buffer candidates are compared pairwise by the VLM; ratings are updated using ELO update rules (e_i ← e_i + T(S_i − E_i)), where E_i is the logistic expected score, absorbing noise across many comparisons rather than trusting any single VLM judgment.
4. Training: Reward is defined as negative Euclidean distance in the latent space of a VAE trained unsupervised on all agent observations: R(s_{t-1}, a_{t-1}) = −‖ψ(o_t) − ψ(o)‖₂, where o is the top-rated candidate image. Every L=5000 steps the target is refreshed and all replay buffer transitions are relabeled.

VLM queries run every K environment steps (K=2000 for Gym, K=500 for Metaworld) with M=5 queries per session. The VAE latent space generalizes reward to unseen states without additional VLM calls.

Key Contributions

• Incremental goal discovery: VLM is queried only to compare image pairs rather than to produce dense per-step rewards, drastically reducing query volume.
• ELO-based noise robustness: Pairwise comparison outcomes are aggregated over many rounds via ELO ratings, preventing single erroneous VLM calls from corrupting the target.
• Unsupervised embedding for reward generalization: VAE trained on unlabeled agent observations provides reward to all states without per-state VLM queries, decoupling feedback frequency from environment step count.
• No environment modifications: Solves tasks with full robot-visible images, unlike RL-VLM-F which removes the robot from frames.
• Periodic reward relabeling: Non-stationary reward from evolving targets is handled by relabeling all replay buffer transitions every L steps.

Results

• Overall: ~95% mean success rate averaged across 7 tasks (2 Gym + 5 Metaworld) vs. ~45% for best competitor (RL-VLM-F).
• Oracle comparison: Nearly matches ground-truth reward (oracle) on all tasks; surpasses it on Drawer Open (oracle fails to learn to pull the handle, GoalLadder does not).
• RL-VLM-F: Solves only easy tasks (CartPole, MountainCar, Drawer Close); fails on harder Metaworld tasks under the same feedback budget.
• VLM-RM / RoboCLIP: Near-zero success on most tasks, consistent with known limitations.
• VLM query efficiency: GoalLadder solves Metaworld tasks after ~4,500 VLM queries on average; PEBBLE (preference-based with ground-truth preferences) requires ~15,000 preference labels for equivalent performance on the same tasks.

Limitations

• Assumes task progress is identifiable from a single image, restricting applicability to static-goal environments; dynamic or partially observable tasks are out of scope.
• Reward is defined by visual feature similarity in VAE latent space, which may be a poor proxy for true task progress in environments where visual appearance and state similarity diverge.
• VAE is a relatively weak visual encoder; more expressive self-supervised representations (e.g., DINO, R3M) are left to future work.
• Evaluation is limited to 7 simulated environments; VLM API costs constrain broader empirical validation.
• Potential inheritance of VLM biases in robotic settings involving humans is flagged but not studied.

Relevance to Vision-Language Models

GoalLadder is a concrete case study in using VLMs as sparse, comparative oracles rather than dense reward generators, exposing a critical design principle: pairwise ranking with noise-aggregation (ELO) is far more robust than direct reward scoring. For researchers tracking VLMs, this work quantifies the gain from treating VLM outputs as noisy votes rather than ground truth, and demonstrates that separating VLM-driven goal identification from learned-embedding reward computation unlocks both robustness and query efficiency. It also highlights the persistent limitation of VLMs in spatial reasoning, which the ELO aggregation partially mitigates by design, and points toward video-capable VLMs as a path to extending the approach to temporally extended goals.