Skip to content

Rank-Then-Act: Reward-Free Control from Frame-Order Progress

🕒 Published (v1): 2026-07-02 08:50 UTC · Source: Arxiv · link

Why this paper was selected

VLM as reward-free ordinal progress scorer for control; directly relevant to agent harness design

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Rank-Then-Act (RTA) trains a VLM offline as a listwise progress scorer on shuffled expert video frames using GRPO, then uses Spearman rank correlation between predicted progress ranks and true temporal indices as the sole reward signal for downstream policy learning — no environment rewards required. The two-stage design decouples reward learning from absolute calibration, yielding a bounded \(\rho \in [-1,1]\) signal that transfers across tasks without per-task tuning.

Problem

Existing video-based reward models for reward-free control suffer three failure modes: (i) chronological inputs allow trivial "later-is-better" shortcuts producing vacuous monotonic scores; (ii) absolute progress scales are ambiguous across tasks/episodes; (iii) online RL requires rewards that are cheap, informative, and robust to distribution shift. Prior VLM reward heads need engineered goal text or per-task calibration, making cross-task reuse impractical.

Method

Stage 1 (Rank): Fine-tune a VLM (Qwen2.5-VL-7B) with LoRA as a listwise progress scorer. Given a \(K\)-frame expert clip, the first frame is fixed as an anchor \(s_\text{anc}\) and the remaining \(K-1\) frames are randomly shuffled. The model generates a reasoning trace and outputs per-frame progress rank \(p_i\). The GRPO objective maximizes the expected Spearman correlation reward: $\(R = \text{spr}(\mathbf{p}, \mathbf{q})\)$ where \(\mathbf{q}\) are ground-truth temporal indices. Shuffling prevents the model from exploiting position cues; anchoring breaks directional ambiguity. The scorer is then frozen.

Stage 2 (Act): During environment interaction, every \(N\) steps construct a sliding window \(W_t = (s_{t-m+1}, \ldots, s_t)\) with the oldest frame as anchor. Draw \(L\) independent permutations of the non-anchor frames, query the frozen scorer for progress ranks under each permutation, compute per-permutation Spearman \(\rho\) between predicted ranks and window timestamps, and average to obtain scalar reward \(r_t\). This is the sole reward fed to PPO (via VL-DAC for discrete tasks) or DrQv2 (continuous control).

Key Contributions

  • A VLM progress scorer trained offline with a GRPO listwise objective on shuffled, anchor-conditioned expert clips, producing ordinal frame ranks invariant to absolute scale.
  • A correlation-only reward primitive: Spearman \(\rho\) between predicted progress ranks and window timestamps, bounded in \([-1,1]\) and requiring no scalar calibration.
  • Empirical demonstration that a single frozen Stage-1 scorer transfers across levels, games, and visual domains (YouTube playthroughs, MetaWorld, COIN AssembleSofa) without per-task fine-tuning.
  • Ablations isolating reward design as the driver of performance gains, including MLP-backbone experiments confirming policy architecture is not the key factor.

Results

  • Catrap Level 2: RTA 1.00 ± 0.00 vs. best baseline Rank2Reward 0.60 ± 0.28; GVL 0.47 ± 0.25; VLM-RM 0.40 ± 0.28.
  • Catrap Level 4: RTA 0.72 ± 0.35 vs. Rank2Reward 0.20 ± 0.00; VLM-RM 0.16 ± 0.20; GVL 0.00.
  • Catrap Level 6: RTA 0.32 ± 0.27 vs. Rank2Reward 0.13 ± 0.09; all other baselines ≤ 0.08.
  • Kirby: RTA is the only method achieving non-zero success (0.07 ± 0.09); all baselines score 0.00.
  • PointMaze-UMaze: RTA (ranking only) outperforms Rank2Reward (ranking only) and Rank2Reward + GAIL throughout training; approaches oracle binary reward.
  • MetaWorld: RTA (ranking only) surpasses R2R (ranking only) across tasks; RTA + GAIL matches or exceeds R2R + GAIL on door-open and door-close.
  • Stage-1 scorer converges to Spearman \(\rho > 0.9\) within 300 steps for nearly all videos.
  • Cross-domain: scorers trained on visually dissimilar sources (MetaWorld, COIN) still enable non-zero success on GameBoy levels.

Limitations

  • On MetaWorld, RTA (ranking only) underperforms GAIL-augmented methods because heavy exploration requirements cause the scorer to occasionally reward dynamically plausible but unsuccessful trajectories (false positives on visual progress).
  • Reward is sparse by design (computed every \(N\) steps), requiring GAE to smooth learning; per-step VLM inference is computationally prohibitive.
  • Very small windows overfit to local temporal consistency; very large windows dilute the signal — window size \(N\) must be tuned per environment.
  • Kirby success rate remains low (0.07), suggesting long-horizon games still challenge the framework despite beating baselines.
  • Experiments rely on Qwen2.5-VL-7B; generalization to smaller or larger VLMs is not systematically evaluated.

Relevance to Vision-Language Models

RTA is a concrete instantiation of VLMs as reward models, demonstrating that a fine-tuned VLM can serve as a task-agnostic ordinal evaluator without any natural-language goal specification or per-task calibration — a direct challenge to the dominant paradigm of goal-conditioned VLM scoring. The GRPO-based listwise training on shuffled frames is a generalizable recipe for teaching VLMs temporal understanding from passive video, relevant to broader efforts on video-VLM reasoning and world models. The finding that a single frozen VLM scorer transfers across visually dissimilar domains strengthens the case for VLMs as universal progress estimators, informing research on VLM-as-critic architectures. For VLM researchers, the correlation-as-reward design offers a principled mechanism to extract grounded, calibration-free signals from pretrained models without instruction tuning on downstream tasks.