Vision-Zero: Scalable VLM Self-Evolution via Multi-Agent Self-Play¶
🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link
Why this paper was selected
ICLR 2026; Vision-Zero: multi-agent self-play for label-free VLM self-evolution
Ask a follow-up
Open an assistant pre-loaded with this paper's context.
TL;DR¶
Vision-Zero is a label-free, multi-agent self-play framework that post-trains VLMs using a "Who Is the Spy?" visual deduction game, requiring no human annotations. Civilians observe a real image while the spy receives a blank input; strategic interaction forces both roles to develop visual reasoning and communication. An iterative algorithm (Iterative-SPO) alternates between self-play and RLVR to prevent equilibrium stagnation, achieving state-of-the-art results on reasoning and vision-centric benchmarks over annotation-heavy baselines.
Problem¶
Existing VLM post-training via SFT, RLHF, or RLVR requires expensive human-annotated multimodal data (e.g., Visual Genome mobilized 33,000 annotators; Ego4D consumed 250,000 annotation hours), capping model capability at human-generated supervision. Self-play eliminates human labeling but has not been effectively applied to VLMs; prior visual games (Sudoku, Gobang) fail to simultaneously satisfy scalability, task alignment, diversity, and label-free data requirements.
Method¶
Vision-Zero instantiates a visual "Who Is the Spy?" game: \(n_c\) civilian agents each receive a real image \(I_c\); one spy agent receives a blank image \(I_s\). Gameplay alternates between:
-
Clue Stage (Self-Play): Each player emits a clue \(u_k \sim \pi_\theta^k(\cdot \mid I_k, h)\) conditioned on role and clue history \(h\). Rewards follow a zero-sum formulation: $\(r_s^{\text{clue}} = -\beta(v_s - \bar{v}_c), \quad r_{c_j}^{\text{clue}} = \frac{\beta}{n_c}(v_s - \bar{v}_c) - \lambda(v_{c_j} - \bar{v}_c)\)$ where \(v_s, v_{c_j}\) are vote counts received. Role-Advantage Estimation (RAE) subtracts an exponential moving average baseline per role to correct for asymmetric information between spy and civilians.
-
Decision Stage (RLVR via GRPO): Civilians output a vote \(\hat{s}_{c_i} \sim q_\theta(\cdot \mid H)\). Discrete rewards are \(+1\) for correct identification, \(-0.5\) for abstention, \(-1\) for wrong vote. Group normalization removes round-level difficulty bias before computing advantages.
Iterative-SPO switches between training stages using hysteresis thresholds on smoothed accuracy \(\bar{\text{acc}}_t\) and n/a rate \(\bar{\text{na}}_t\): when civilians identify the spy too easily (Decision saturates), training shifts to the Clue stage to increase difficulty; when identification becomes too hard (clue stage equilibrium), training reverts to the Decision stage. The per-iteration loss is \(\mathcal{L}_t = m_t \mathcal{L}^{\text{clue}}(\theta) + (1-m_t)\mathcal{L}^{\text{dec}}(\theta)\).
Data inputs are label-free and domain-agnostic: 2,000 CLEVR synthetic images, 1,000 ChartQA images, and 1,000 real-world images from ImgEdit — all used without task-specific annotations.
Key Contributions¶
- Vision-Zero: first gamified self-play post-training framework for VLMs with zero human annotation in the loop, supporting arbitrary image inputs.
- Iterative-SPO: novel two-stage alternating algorithm combining self-play (zero-sum, role-advantage-estimated rewards) and RLVR (GRPO with group normalization) to avoid equilibrium stagnation and knowledge plateau.
- Demonstration that a single game environment generalizes across three distinct image domains (synthetic, chart, real-world) without task-specific engineering.
- Empirical evidence that label-free self-play surpasses annotation-heavy baselines across reasoning, chart QA, and vision-centric benchmarks.
Results¶
All results are post-training on Qwen2.5-VL-7B (baseline: 41.1 avg across 6 benchmarks):
- VisionZero (CLEVR): 44.3 avg — MathVista 72.2, MathVision 28.4, WeMath 39.2, MathVerse 53.2, LogicVista 49.8, DynaMath 22.9
- VisionZero (Real-World): 44.5 avg — MathVista 73.1, MathVision 28.5, WeMath 40.1, MathVerse 52.1, LogicVista 50.8, DynaMath 22.5
- Outperforms MM-Eureka-Qwen-7B (42.9), VLAA-Thinker-7B (41.9), ViGaL-Snake+Rotation (43.0), and OpenVLThinker-7B (40.9)
- Approaches proprietary Gemini 2.0 Flash (54.4 avg) while being a 7B open model
- Sustained +3% gain on MathVision validation set vs. base model (vs. 0% for Gobang-trained models)
- GPT-based evaluation of spy reasoning traces shows improvements in PLANNING, RETRIEVE, DECOMPOSE, and STRATEGIZE scores after training
Limitations¶
- Only validated on Qwen2.5-VL-7B; generalization to other model families not demonstrated in the provided text.
- CLEVR data required ~6 hours of GPU rendering on an A100; real-world/chart data required manual dataset selection, so the pipeline is not entirely zero-cost.
- Iterative-SPO introduces several sensitive hyperparameters (\(\alpha, \beta, \lambda, \rho, \tau_{\text{acc}}^\uparrow, \tau_{\text{na}}^\downarrow\), dwell time \(K_{\min}\)) whose tuning is not characterized in the available text.
- The game environment is inherently binary (spy vs. civilian), which may limit the variety of reasoning patterns it can elicit compared to open-ended task supervision.
- Pure self-play is known to converge to local equilibria; Iterative-SPO mitigates but does not provably eliminate this risk.
Relevance to Vision-Language Models¶
Vision-Zero directly addresses the data scalability bottleneck for VLM post-training: by replacing human-curated annotations with autonomous game-generated signals, it decouples capability growth from annotation budgets. The Iterative-SPO paradigm is a significant methodological contribution to VLM RL training, complementing concurrent work like RLVR (DeepSeek-R1-style) and gamified reasoning (SPIRAL, Absolute Zero) while being the first to operate in the visual modality without task-specific labels. For researchers tracking VLMs, this paper establishes that competitive multi-agent self-play over raw images is a viable and generalizable post-training signal, with direct implications for scaling VLM capabilities beyond the human knowledge ceiling.