Skip to content

Vision-Zero: Scalable VLM Self-Evolution via Multi-Agent Self-Play

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

RL self-play without manual data for VLM reasoning capability improvement

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Vision-Zero is a label-free, multi-agent self-play framework that trains VLMs by having them compete in a visual "Who Is the Spy?" game generated from arbitrary images. It introduces Iterative Self-Play Policy Optimization (Iterative-SPO), which alternates between self-play and RLVR to prevent equilibrium stagnation. Despite requiring no human annotations, it outperforms annotation-based post-training methods across reasoning, chart QA, and vision-centric tasks.

Problem

VLM post-training suffers from two bottlenecks: (1) prohibitive annotation cost (e.g., COCO Attributes: $60,480 for 200K objects; Ego4D: 250K annotation hours) limiting data scale and diversity, and (2) a knowledge ceiling where model capability is fundamentally bounded by human-generated supervision. Existing self-play methods for LLMs (SPIRAL, Absolute Zero) target text-only tasks and do not transfer to multimodal settings; visual game environments (e.g., Gobang/Sudoku) fail to simultaneously satisfy all four desiderata: task-skill alignment, scalable difficulty, domain diversity, and minimal external data.

Method

Vision-Zero casts VLM post-training as a competitive visual social-deduction game. \(n_c\) civilian agents each receive image \(I_c\); one spy agent receives a blank image \(I_s\). Play proceeds in two stages per round:

  • Clue Stage (Self-Play): All players sequentially emit one-sentence descriptions \(u_k \sim \pi_\theta(\cdot | I_k, h)\); history \(h\) is visible but reasoning is private. A zero-sum reward penalizes votes received: \(r_s^{\text{clue}} = -\beta(v_s - \bar{v}_c)\) for the spy and \(r_{c_j}^{\text{clue}} = \frac{\beta}{n_c}(v_s - \bar{v}_c) - \lambda(v_{c_j} - \bar{v}_c)\) for civilians. Role-Advantage Estimation (RAE) corrects for asymmetric information by subtracting an exponential moving average baseline per role.

  • Decision Stage (RLVR/GRPO): Civilians vote on who is the spy; reward is \(+1\) for correct vote, \(-0.5\) for abstain, \(-1\) for wrong. Group normalization across the civilian batch removes round-level difficulty variance before GRPO advantage weighting.

Iterative-SPO switches training phases using hysteresis thresholds on smoothed accuracy \(\bar{\text{acc}}_t\) and abstention rate \(\bar{\text{na}}_t\): when spy identification becomes too easy (clue saturation), shift to clue-stage self-play; when identification becomes too hard (decision saturation), shift back to RLVR. The combined per-iteration loss is \(\mathcal{L}_t = m_t \mathcal{L}^{\text{clue}}(\theta) + (1 - m_t) \mathcal{L}^{\text{dec}}(\theta)\).

Input images are entirely label-free; three dataset types are validated: CLEVR synthetic scenes (2,000 rendered images), ChartQA samples (1,000), and real-world ImgEdit images (1,000).

Key Contributions

  • First gamified, zero-human-in-the-loop VLM post-training framework operating on arbitrary unlabeled images.
  • Iterative-SPO: a two-stage alternating algorithm combining self-play (zero-sum clue-stage reward + RAE) with RLVR (GRPO decision-stage reward), preventing both equilibrium collapse and knowledge saturation.
  • Demonstrated domain agnosticism: single framework trained on CLEVR, chart, or real-world images generalizes across diverse benchmarks without task-specific data engineering.
  • Role-Advantage Estimation to correct information asymmetry between spy and civilian roles during self-play.

Results

All models post-trained on Qwen2.5-VL-7B base. Baselines include MM-Eureka, VLAA-Thinker, OpenVLThinker, ViGaL (annotation-based).

Reasoning/Math (Table 1): - VisionZero-Qwen-7B (Real-World): avg. 44.5 across six benchmarks vs. 43.0 for best annotated baseline (ViGaL-Snake+Rotation); improves MathVista from 68.2 → 73.1, MathVerse 36.1 → 40.1, LogicVista 47.2 → 50.8. - Surpasses GPT-4o average (43.4) with a 7B model.

Chart/Doc QA (Fig. 2): - VisionZero-Chart: +86.1% relative gain on ChartQA, +94.8% on DocVQA over base Qwen2.5-VL-7B.

Vision-Centric: - +68.1% on RealWorldQA, +55.2% on BLINK over base.

Ablation (Fig. 3, bottom-left): - Vision-Zero achieves +3% on MathVision validation over base; Gobang game environment yields no improvement, confirming the importance of domain-agnostic visual game design.

Limitations

  • Experiments are confined to one base model family (Qwen2.5-VL-7B); generalization to other VLM architectures is not demonstrated.
  • The spy game mechanic requires paired images with visual differences; constructing meaningful image pairs for highly abstract or domain-specific content may be non-trivial.
  • Self-play still requires GPU compute for game rollouts (CLEVR rendering alone took ~6 hours on A100); true cost vs. annotation-based methods is not systematically compared.
  • Hysteresis thresholds \(\tau^{\uparrow}_{\text{acc}}, \tau^{\downarrow}_{\text{na}}\), etc. in Iterative-SPO are hyperparameters whose sensitivity is not ablated in the provided text.
  • Paper text is truncated; full results on some baselines (ViGaL-Snake, ViGaL-Rotation) are taken from original papers under potentially different evaluation conditions.

Relevance to Agentic AI / LLM Agents

Vision-Zero is a concrete instance of multi-agent self-improvement via competitive game-play, directly relevant to the line of work (SPIRAL, Absolute Zero) exploring how agents can bootstrap capability without human supervision. The Iterative-SPO mechanism—dynamically switching between self-play and verifiable-reward RL based on detected equilibrium signals—is a practically important technique for sustaining improvement in long-running agentic training loops that would otherwise converge prematurely. The asymmetric-role structure (spy vs. civilians) operationalizes theory-of-mind reasoning (inferring hidden information from others' utterances) as an emergent training objective, which is increasingly relevant to agent communication and deception-detection research. The label-free, domain-agnostic design principle shows how agentic game environments can be constructed from raw observations rather than engineered reward functions, with implications for scalable self-play in embodied and tool-use agent settings.