Skip to content

Tandem Reinforcement Learning with Verifiable Rewards

🕒 Published (v1): 2026-06-26 15:00 UTC · Source: Arxiv · link

Why this paper was selected

Tandem RLVR: Robert West (EPFL); extends verifiable-reward RL to weaker models—critical scalability question

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Tandem Reinforcement Learning (TRL) extends the tandem training paradigm into GRPO-style RLVR by having a trainable senior model stochastically co-generate rollouts with a frozen junior (its own pre-RL base), rewarding the team jointly. Training Qwen3-4B-Instruct on competition math, TRL matches vanilla GRPO on solo reasoning while simultaneously improving multi-model handoff robustness, reducing distributional drift, and increasing chain-of-thought legibility to the junior — all from a single change to rollout structure.

Problem

RLVR-trained models drift toward idiosyncratic token patterns (language mixing, steganographic reasoning, syntactic irregularities) that weaker partner models and human overseers cannot predict or continue, undermining multi-model pipelines and scalable oversight. Existing defenses (KL penalties against a reference policy, supervised distillation, process supervision) require committing upfront to an explicit specification of "intelligible" reasoning, which is deployment-context-dependent and hard to write down in practice. Prior tandem training proofs-of-concept (chess, simplified GSM8K) left open whether the paradigm scales into the long chain-of-thought regime of modern RLVR.

Method

TRL initializes both a trainable senior \(\pi_\text{sen}\) and a frozen junior \(\pi_\text{jun}\) from the same base model. At each rollout, the active author alternates stochastically at word boundaries: a fair coin (\(p = 0.5\)) is flipped at every orthographic word onset to determine which model emits the next word. Both models forward on the full shared context \(x \cdot y_{<t}\); the active model's token \(y_t\) is appended and fed back to both. The completed response receives a single binary verifier reward \(r(y) \in \{0,1\}\) on the boxed final answer. The senior is updated with standard GRPO:

\[\mathcal{L}_\text{TRL}(\text{sen}) = -\mathbb{E}\!\left[\sum_{t:\,a_t=\text{sen}} A_t \log \pi_\text{sen}(y_t \mid x \cdot y_{<t})\right]\]

where \(A_t\) is the GRPO group-relative advantage and the mask selects only senior-emitted tokens. The junior-imitation coefficient \(\lambda_\text{jun}\) is set to 0, so differences from vanilla GRPO are attributable solely to rollout structure. Word-level granularity is chosen over token-level (disrupts coherent utterances) and sentence-level (senior can suppress end-of-sentence tokens to game the schedule).

Key Contributions

  • Introduces TRL, the first instantiation of tandem training inside a full RLVR/GRPO pipeline with long chain-of-thought.
  • Demonstrates that three compatibility properties (handoff robustness, distributional anchoring, junior legibility) emerge jointly from a single rollout-structure change, with no modification to reward, verifier, or loss.
  • Identifies word-level handoff boundaries as the coarsest granularity the senior cannot trivially erase, closing a reward-hacking loophole present at sentence-level.
  • Shows that adding a per-token KL penalty toward the junior on top of vanilla GRPO (an ablation) does not reproduce TRL's gains, establishing that rollout co-generation is the operative mechanism rather than regularization.
  • Proposes rollout structure as an underexplored design axis in the RLVR pipeline.

Results

  • Solo reasoning (pass@k): TRL tracks vanilla GRPO within run-to-run variance on AMC 23–25, AIME 24–26, and Minerva Math across \(k \in \{1,\ldots,32\}\); no consistent direction favoring either model.
  • Handoff robustness (reasoning-step schedule at inference): TRL senior paired with frozen junior outperforms GRPO senior paired with junior by +4.6 pass@1 and +6.6 pass@8 on AIME (29.3 vs. 24.7 at \(k=1\); 44.4 vs. 37.8 at \(k=8\)); macro-average gap grows from +2.2 at \(k=1\) to +3.1 at \(k=8\).
  • Distributional drift: TRL is 14% lower KL divergence from the base (0.022 vs. 0.026 nats); among the 500 tokens most displaced by GRPO, 87% shift back toward base under TRL (Spearman \(\rho = -0.58\), \(p < 10^{-275}\)).
  • Junior legibility (cross-entropy): TRL chain-of-thought is 0.010 nats lower cross-entropy under the junior on average (0.122 vs. 0.132), reaching 17% reduction in per-token surprisal on Minerva; token-level distribution overlap improves by ~30% reduction in shortfall from perfect agreement (0.973 vs. 0.960).

Limitations

  • Experiments are limited to a single base model (Qwen3-4B-Instruct) and a single domain (competition math); generalization to other modalities, domains, and model scales is unverified.
  • The junior is always the senior's own pre-RL base; asymmetric pairings (different model families or sizes) are noted as available but not evaluated.
  • Checkpoint selection is done on tandem-rollout accuracy rather than solo accuracy, making direct comparison with GRPO training curves require separate axes; the selection protocol may favor TRL in tandem-specific metrics.
  • Wall-clock overhead from running two forward passes per token is reported (approximately 9.4h vs. 7.8h for GRPO based on Figure 5), but scaling implications at larger model sizes are not analyzed.
  • The paper does not address how TRL behaves when junior and senior diverge substantially after many training steps, or whether the frozen junior becomes a bottleneck as the senior's capability improves.

Relevance to Agentic AI / LLM Agents

TRL directly targets the multi-model communication problem central to agentic pipelines: when one LLM agent hands off a reasoning trace to a weaker partner (drafter-target inference, human-AI collaboration, hierarchical agent systems), the stronger model's internal reasoning must remain within the interpretive reach of the downstream agent. By operationalizing "intelligibility" as team success under stochastic handoffs rather than as a pre-specified reference distribution, TRL provides a scalable, reward-driven mechanism for training agents that stay compatible with weaker collaborators without sacrificing capability. This connects to scalable oversight research — the result that TRL reduces distributional drift and steganographic chain-of-thought patterns is directly relevant to monitor-based oversight architectures where a weaker model must verify a stronger one's reasoning. More broadly, TRL demonstrates that rollout structure (who generates which tokens, not just what reward is given) is a controllable design axis for shaping emergent multi-agent behavior, opening a new lever for agentic system designers.