Skip to content

Automating Potential-based Reward Shaping with Vision Language Model Guidance

🕒 Published (v1): 2026-06-25 15:45 UTC · Source: Arxiv · link

Why this paper was selected

Automates reward shaping via VLM potential functions; practical RL agent improvement without dense human annotation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLM-PBRS replaces hand-crafted potential functions in potential-based reward shaping (PBRS) with a preference model trained on VLM-generated labels over image pairs. Because PBRS's policy-invariance guarantee holds for any potential function—correct or not—the method tolerates noisier labels from cheaper, smaller VLMs while still accelerating sample efficiency without risking reward hacking.

Problem

Sparse-reward RL suffers from credit-assignment failures and poor exploration; naive reward shaping risks reward hacking (policy optimizes auxiliary signal instead of true objective). PBRS avoids hacking but requires an expert-crafted potential function \(\Phi(s)\). Prior VLM-based reward learning (e.g., RL-VLM-F) uses the learned reward model directly, which demands high labeling accuracy and expensive large VLMs; errors there corrupt the policy objective.

Method

VLM-PBRS learns \(\Phi\) from VLM preference labels without hand-crafted state grounding:

  1. Preference labeling: Uniformly sample image pairs from the replay buffer; query a small open-weight VLM with a structured prompt (describe common elements → describe goal-relevant differences → output label \(y \in \{0, 1, -1\}\)) using only a natural-language goal description.
  2. Potential model training: Fit a parameterized preference function \(r_\psi\) via Bradley-Terry cross-entropy loss (Eq. 3). After each update, recompute shaping rewards retroactively in the replay buffer.
  3. Potential function: Wrap \(r_\psi\) through a sigmoid and scale: \(\Phi(s) = \lambda\sigma(r_\psi(s))\) for non-terminal states, \(\Phi(s)=0\) at terminals (required for policy invariance). \(\lambda=0.9\) keeps shaped goal reward positive.
  4. Shaped reward: \(R'(s,a,s') = R(s,a,s') + \gamma\Phi(s') - \Phi(s)\).
  5. RL backbone: SAC with the shaped reward. Labeling and potential updates run every \(K\) iterations interleaved with policy improvement.

Key simplification over RL-VLM-F: single VLM call per pair (no second LLM call to parse output), extracting \(y\) directly with "the answer is \(y\)" in the prompt.

Key Contributions

  • First application of VLM preference-based learning to synthesize a PBRS potential function.
  • Formal argument that PBRS policy invariance relaxes label accuracy requirements, enabling small/cheap VLMs where direct reward modeling would fail.
  • Single-prompt labeling pipeline (vs. RL-VLM-F's two-stage VLM+LLM pipeline), reducing inference cost.
  • Empirical validation across 7 tasks in Meta-World (door-open, window-open, drawer-open, button-press) and Franka Kitchen (microwave, light-switch, top-burner).
  • Analysis of the relationship between VLM label accuracy and sample efficiency gain.

Results

(Paper text truncated before quantitative tables; the following is stated qualitatively in the text.) - VLM-PBRS improves sample efficiency over the sparse-reward baseline across Meta-World and Franka Kitchen tasks. - Even with lower label accuracy from smaller VLMs, PBRS prevents policy degradation—in contrast to RL-VLM-F where noisy labels can bias the learned policy. - MAE loss (robust to noisy labels) showed no improvement over cross-entropy in ablation (Section 6.5). - Franka Kitchen (more visual clutter, harder grounding) required an extended system prompt to achieve sufficient label accuracy.

Limitations

  • Specific accuracy thresholds below which PBRS guidance becomes ineffective are environment-dependent and not derived analytically.
  • Label quality depends on VLM visual reasoning; distractor objects (Franka Kitchen) degrade accuracy, requiring task-specific prompt engineering.
  • Method still requires a natural-language goal description—automated goal elicitation is out of scope.
  • Quantitative results are in truncated sections; the relationship between label accuracy and sample efficiency gain may not generalize across VLM sizes or environment types.
  • Evaluation is limited to tabletop manipulation; generalization to visually diverse or partially observable domains is untested.

Relevance to Agentic AI / LLM Agents

This work is directly relevant to reward design for embodied agents: it shows how VLMs can automate a previously manual bottleneck (potential function engineering) while providing theoretical guarantees that noisy VLM outputs cannot corrupt the agent's learned objective. For LLM/VLM-augmented agent pipelines, the insight that PBRS's policy-invariance absorbs label noise is practically important—it decouples "how good is the VLM?" from "will the agent converge to the right behavior?", enabling cost-effective deployment of smaller models. This connects to the broader trend of using foundation models as reward or preference oracles (RLHF, RLAIF, VLM-F) while mitigating their failure modes through theoretically grounded RL primitives.