GTR-Turbo: Merged Checkpoint is Secretly a Free Teacher for Agentic VLM Training¶

🕒 Published (v1): 2025-12-15 07:11 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

GTR-Turbo eliminates the dependency on expensive external teacher models (GPT-4o, Gemini) in multi-turn VLM agent RL training by using TIES-merged historical checkpoints from the ongoing training run as a "free" teacher. It matches or surpasses the original GTR framework while cutting wall-clock training time by ~50% and compute cost by ~60%.

Problem¶

Multi-turn RL for VLM agents suffers from sparse rewards and long-horizon credit assignment, leading to "thought collapse" (repetitive, incoherent outputs). Prior process-guidance methods like GTR densify rewards using external teacher models (GPT-4o, Gemini), but these impose prohibitive API costs (~$150–$300 per training run), introduce network latency, create data-privacy risks, and are sometimes inaccessible, severely limiting reproducibility and scalability.

Method¶

At each RL update step, GTR-Turbo saves the current checkpoint into a rolling buffer. It then merges all buffered checkpoints using TIES-merging (trim low-magnitude parameter deltas, elect a consensus sign per parameter via majority vote, average only sign-consistent parameters) to produce a composite teacher model that is strictly stronger and more stable than any individual checkpoint. This merged teacher guides subsequent PPO training via one of two objectives:

GTR-Turbo (SFT): the merged teacher autoregressively generates reference "thought" tokens; an SFT loss on those tokens is added to the PPO objective (replicating the GTR mechanism without an external model).
GTR-Turbo (KL): a single forward pass through the merged teacher yields reference logits; the negative reverse-KL divergence between agent and teacher token distributions is used as an auxiliary thought reward in PPO, with negative values clipped. This eliminates autoregressive teacher generation, halving compute overhead further.

Weight-averaging strategies (SMA or EMA with tunable α) control checkpoint weighting; α=0.5 EMA or SMA both work well.

Key Contributions¶

Identifies that TIES-merged checkpoints from an in-progress RL run consistently outperform the current training snapshot, making them viable self-referential teachers at zero extra training cost.
Two guidance variants (SFT and KL) that allow trading off compute efficiency vs. guidance quality, with the KL variant requiring only one forward pass per step.
Removal of all external API dependencies, making the entire pipeline locally deployable and privacy-preserving.
Demonstrates that clipped reverse-KL is the most stable estimator for token-level distillation in this dynamic teacher–student setup.
Validates across two diverse benchmarks (Points24 card game, ALFWorld embodied navigation) using Qwen2.5-VL-7B and Qwen3-VL-8B backbones.

Results¶

Points24: GTR-Turbo (KL) achieves 53.5% success rate vs. GTR's 44.5% and RL4VLM's 3.5%; episode return 2.39 vs. 0.53 (GTR) and −13.3 (RL4VLM); outperforms GPT-4o (2.5%) and Qwen2.5-VL-72B (5.6%).
ALFWorld: GTR-Turbo (KL) reaches 15% average success rate, on par with GTR (16%) and far above RL4VLM (8%).
Training time: GTR-Turbo (KL) takes 89h on Points24 vs. 191h for GTR (≈53% reduction); 78h vs. 164h on ALFWorld.
Cost: GTR-Turbo (KL) ≈$115 (GPU cost) vs. $308 (GPT-4o API) on Points24; eliminates 100% of external API token usage.
TIES merging outperforms plain linear averaging; SMA and balanced EMA (α=0.5) perform comparably; all non-negative KL estimators improve over the raw K1 estimator, with clipping best.

Limitations¶

Self-improvement fails when the base model is too weak (initial success rate <5%); a strong external teacher is still required in such cases.
Experiments are restricted to 7B-scale models; scaling behavior at larger sizes is unverified.
Cost estimates for GPU-based teacher deployment are volatile and depend on market pricing.
ALFWorld results show GTR-Turbo (SFT) underperforms GTR on several sub-tasks (Clean, Heat) despite comparable averages, suggesting the merged teacher provides weaker guidance on highly diverse task types.

Relevance to Agentic AI / LLM Agents¶

GTR-Turbo addresses a core bottleneck in self-improving agent training: the reliance on privileged external oracles for dense process supervision, which has been a hidden scalability ceiling for multi-turn RL pipelines. By demonstrating that a model's own training trajectory encodes sufficient supervisory signal—extractable via checkpoint merging—the work establishes a practical, closed-loop paradigm for agent self-improvement without proprietary API access. This connects directly to the broader research thread on autonomous agent training (GRPO, DAPO, on-policy distillation, PPO-based VLM fine-tuning) and suggests that model-merging techniques from continual learning can serve as a lightweight substitute for process reward models or LLM-as-judge oracles in long-horizon agentic settings.