Skip to content

PRIMT: Preference-based Reinforcement Learning with Multimodal Feedback and Trajectory Synthesis from Foundation Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PRIMT is a preference-based reinforcement learning (PbRL) framework that replaces human annotators with foundation models (LLMs + VLMs) for generating synthetic preference feedback over robot trajectories. It addresses two core PbRL failure modes—early-stage query ambiguity and coarse credit assignment—via foresight trajectory bootstrapping and counterfactual hindsight augmentation. Evaluated on 8 locomotion/manipulation tasks, PRIMT consistently outperforms single-modality FM baselines and is competitive with oracle reward functions.

Problem

Prior FM-driven PbRL methods rely on a single modality: LLM-based methods use textual state-action projections (prone to hallucination of spatial events) while VLM-based methods use visual renderings (missing temporal dynamics). Neither resolves two intrinsic PbRL challenges: (1) query ambiguity—early-stage trajectory pairs are uniformly low-quality, yielding uninformative preference queries; and (2) credit assignment—trajectory-level preference labels cannot be reliably attributed to specific state-action steps.

Method

PRIMT has two core modules:

1. Hierarchical Neuro-Symbolic Preference Fusion - Trajectory preprocessing: Textual projections (dimension-specific state-action arrays) for LLM evaluation; hybrid keyframe extraction (near-zero velocity, smoothing residual peaks, change-point detection) for VLM evaluation. - Intra-modal fusion: Each modality (LLM/VLM) is queried K times with permuted orderings (crowd-check); majority vote yields label; confidence is a weighted mix of average confidence among agreeing votes and label consistency ratio. - Inter-modal fusion via PSL: Four weighted first-order logic rules (agreement, two conflict-resolution rules, indecision) are instantiated with continuous atoms—visual discriminability (CLIP+Wasserstein distance over keyframe sets), temporal discriminability (trajectory volatility via second-order finite differences), and per-modality confidence—then solved as a convex hinge-loss MRF optimization to infer the final soft preference label.

2. Bidirectional Trajectory Synthesis - Foresight generation: LLMs generate diverse, task-aligned trajectories via a three-step CoT (action plan → code snippets → execution under varied conditions) before training begins; these bootstrap the trajectory buffer as preference anchors, not assumed-optimal demonstrations. - Hindsight augmentation: During training, when a clear preference is detected, LLMs apply SCM-based counterfactual reasoning: (i) abduct causal steps from the preferred trajectory, (ii) minimally intervene at a key step to generate a sub-preferred counterfactual, (iii) verify via LLM that the preference reverses. Verified counterfactuals are paired with the original for reward learning under a causal auxiliary loss (Eq. 15): contrast loss at edited steps + consistency loss at unchanged steps. Final loss: L_final = L_pref + λ_cf · L_aux_cf.

Key Contributions

  • PRIMT: a general zero-shot PbRL framework using FMs as both passive evaluators and active trajectory synthesizers, eliminating human annotation.
  • Hierarchical neuro-symbolic fusion combining VLM and LLM modalities via PSL-based inter-modal reasoning, handling uncertainty and cross-modal conflicts.
  • Foresight trajectory generation (code-generation CoT) to warm-start the trajectory buffer and reduce early-stage query ambiguity.
  • Hindsight counterfactual augmentation with SCM-based causal reasoning + causal auxiliary loss for precise state-action-level credit assignment.
  • Real-world validation on a Kinova Jaco robot (block lifting and stacking).

Results

  • Main comparison (8 tasks: Hopper Stand, Walker Walk, Button Press, Door Open, Sweep Into, PickSingleYCB, StackCube, PegInsertionSide): PRIMT consistently outperforms RL-VLM-F (VLM-only), RL-SaLLM-F (LLM + self-augmentation), PrefCLM (crowdsourced LLMs), and PrefMul (naive multimodal FM input).
  • PRIMT is competitive with PrefGT (oracle scripted reward), even surpassing it on Sweep Into and PegInsertionSide; all other FM-based baselines fall clearly below oracle.
  • Ablations on Door Open and PickSingleYCB: removing inter-modal fusion causes the largest performance drop; removing hindsight augmentation or the causal auxiliary loss significantly reduces final task performance; removing foresight generation slows early-stage learning without reducing final performance.
  • Foresight generation reduces FM indecision rates and improves preference label accuracy vs. baselines.
  • Reward alignment (visual analysis): PRIMT's learned reward tracks ground-truth task progress more closely than baselines/ablations.
  • Cost efficiency: GPT-4o-mini backbone achieves 94% cost reduction with 16–26% performance drop → 13Ă— better cost-performance efficiency.

Limitations

  • Increased inference cost from multimodal FM queries (LLM + VLM crowd-checks per preference query); detailed cost analysis in appendix.
  • Evaluated only in single-agent robotic settings; extension to multi-agent and non-robotic domains is future work.
  • Foresight generation quality is task-dependent; LLMs may struggle with high-dimensional manipulation action spaces (acknowledged via RL-SaLLM-F comparison on ManiSkill).
  • Counterfactual quality depends on LLM capability; smaller backbones incur a 16–26% performance penalty.
  • PSL rule weights are fixed/hand-defined; learning them from data could improve adaptability.

Relevance to Vision-Language Models

VLMs are a core component of PRIMT's feedback pipeline: they evaluate keyframe sequences from robot trajectories to provide spatially grounded preference signals that LLMs (operating on textual projections) miss. The paper directly benchmarks VLM-only feedback (RL-VLM-F) as a baseline and shows that VLMs alone are insufficient due to neglected temporal dynamics, motivating the multimodal fusion architecture. For VLM researchers, PRIMT provides a structured recipe—keyframe extraction, crowd-check querying, CLIP-based visual discriminability—for deploying VLMs as reliable reward signals in embodied settings. The PSL-based inter-modal fusion pattern (combining VLM spatial reasoning with LLM temporal reasoning) is transferable to other VLM-in-the-loop decision-making pipelines.