A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism¶

🕒 Published (v1): 2026-07-14 11:17 UTC · Source: Arxiv · link

Why this paper was selected

Controlled null result on GRPO failing for small VLMs; mechanistic analysis; essential calibration for agent RL fine-tuning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT ✦ Ask Claude

TL;DR¶

GRPO fails to improve a competent 4B–8B vision-language web agent on tasks it has largely mastered, and moderate-to-high learning rates make it credibly worse. The null is not a broken pipeline: the identical harness gains +22 points on tasks where the sampled policy already beats greedy, revealing a headroom condition that predicts both success and failure. A weight-grafting mechanistic analysis shows the degrade regime localizes to attention/MLP blocks while large embedding drift is causally inert.

Problem¶

It is unclear whether GRPO adds genuinely new skill to a small (4B–8B) language or vision-language model web agent, or merely sharpens behavior the supervised checkpoint already has. Prior work shows RL can help web agents but does not settle this question for agents already competent on their task distribution—the regime most deployed systems occupy.

Method¶

The authors fix a single agent (Qwen3-VL 4B/8B), environment (MiniWoB), reward (sparse binary terminal signal broadcast to every step: $r(\tau) = \mathbf{1}[\text{success}(\tau)]$), and task set, then sweep 18 GRPO configurations varying learning rate, KL weight $\beta$, seed, initialization, and clip bound $(\epsilon_{lo}=0.20, \epsilon_{hi}=0.28)$.

GRPO objective. The advantage uses group mean-centering without standard-deviation normalization: $$A_i = r_i - \frac{1}{G}\sum_{j=1}^G r_j$$ The clipped surrogate (asymmetric) is: $$\mathcal{L}_\text{GRPO} = -\frac{1}{C}\sum_c \min\!\bigl(\rho_c A,\;\text{clip}(\rho_c, 1-\epsilon_{lo}, 1+\epsilon_{hi})\,A\bigr)$$

Three experiments are run: (A) controlled null—18-run grid on mastered tasks; (B) positive control—same recipe on tasks where $\Delta_\text{head} = \text{SR}_\text{sample} - \text{SR}_\text{greedy} > 0$; (C) mechanism—fixed-state effective rank probes (erank, PR, stable rank), output-distribution probes (argmax agreement, entropy, degeneracy), and causal weight grafting that restores one component group (attention, MLP, or embedding) from initialization and re-evaluates success rate.

Evaluation uses clean greedy decoding, Wilson score intervals, exact paired McNemar tests, and task-clustered bootstrap on $\Delta\text{SR}$.

Key Contributions¶

Controlled null: across 18 runs and multiple robustness checks (25 eval seeds, 6 training seeds, Set-of-Marks modality, 8B scale), no GRPO configuration credibly improves a strong SFT baseline; higher learning rates credibly reduce performance.
Headroom condition: GRPO improves the agent only when $\Delta_\text{head} > 0$ (sampled SR > greedy SR); the +22-point gain on the headroom set with the same harness rules out a broken pipeline and pins the null on task structure.
Double dissociation: the degrade regime ($\text{LR}=10^{-5}$, $-15$ pp) destroys late-layer effective rank while earlier layers remain intact; the collapse regime ($\text{LR}=2\times10^{-5}$, $-49$ pp) preserves rank but drives argmax agreement to zero—two distinct failure modes.
Causal localization via grafting: restoring attention or MLP blocks from initialization recovers frontier success from ~11% to ~37–40%; restoring the embedding (largest raw weight drift) leaves success near 14%, showing embedding movement is a correlational red herring.
Scale-dependent rank–capability coupling: at 4B, late-layer effective rank tracks capability bidirectionally; at 8B this coupling breaks down.

Results¶

Mastered grid (4B, text): SFT baseline 49.1%; best GRPO arm 52.7% ($\Delta=+3.6$, 95% CI $[+0.0, +10.9]$, McNemar $p=0.50$)—no credible gain.
Degrade regime ($\text{LR}=10^{-5}$): SR 33.3%, $\Delta=-15.0$ pp (CI $[-20.9, -9.5]$, $p\leq0.04$).
Collapse regime ($\text{LR}=2\times10^{-5}$): SR 0.0%, $\Delta=-49.1$ pp (CI $[-62.7, -36.4]$, $p<0.001$).
Positive control (headroom set, 4B, text): SFT 20.0%, best GRPO 42.0%, $\Delta=+22$ pp (CI $[+8, +40]$, $p=0.007$); 5 of 6 configurations rise 12–22 pp credibly.
Set-of-Marks track: baseline raised to 63.6%; still no credible GRPO gain.
8B backbone: same qualitative pattern—no credible gain at low LR, credibly worse at high LR.
Grafting (degrade regime): attention restore → 37% frontier SR; MLP restore → 40%; embedding restore → ~14%; random matched-parameter-count restore → mean 19%, 95th percentile 23%.

Limitations¶

Single benchmark (MiniWoB), chosen for determinism; results may not transfer to noisier, open-ended benchmarks like WebArena.
Text-track learning-rate monotone shape (low→flat, mid→degrade, high→collapse) is specific to 4B; at 8B and Set-of-Marks the shape differs.
At 8B, effective rank dissociates from capability, so the mechanistic account derived at 4B does not generalize to larger scale.
Task mastery is defined on a fixed 11-task grid; the headroom condition is measured post-hoc on the supervised policy, requiring an extra profiling step before training.
No curriculum or online data-collection strategies tested; results are specific to offline GRPO on a fixed task distribution.

Relevance to Vision-Language Models¶

This paper provides a rigorous controlled study of GRPO post-training for VLMs acting as web agents—a setting increasingly relevant as VLMs like Qwen3-VL are deployed in agentic pipelines. The key practical finding is that the sampled-vs-greedy headroom gap ($\Delta_\text{head}$) is a cheap pre-training diagnostic: VLM practitioners can profile their supervised checkpoint before committing GRPO compute, saving cost when greedy already dominates sampling. The mechanistic dissociation between attention/MLP failure and embedding drift challenges the common assumption that large parameter movement implies causal responsibility—a methodological lesson applicable to any RL fine-tuning of VLMs. The scale-dependent breakdown of the rank–capability coupling at 8B signals that interpretability tools calibrated on smaller VLMs may not transfer directly to larger ones.