Skip to content

ViPER: Empowering the Self-Evolution of Visual Perception Abilities in Vision-Language Models

đŸ•’ Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; ViPER: self-evolution for fine-grained visual perception in VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

đŸ’¬ Ask ChatGPT✦ Ask Claude

TL;DR

ViPER is a self-bootstrapping framework that improves fine-grained visual perception in VLMs through a two-stage coarse-to-fine pipeline: caption self-refinement (global scene understanding) followed by visual-operation prediction (local, fine-grained change detection). It uses a closed loop of VLM-driven data synthesis via diffusion models and two-stage GRPO-based reinforcement learning, requiring no external supervision or cold-start data. Applied to Qwen2.5-VL-3B/7B, it produces the Qwen-Viper series with consistent perception improvements across seven benchmarks.

Problem

VLMs have a critical bottleneck in fine-grained visual perception. Existing post-training strategies are inadequate: SFT via distillation from stronger models is costly and harms generalization; RFT with tool-augmented "thinking-with-image" methods improves system-level outcomes but emphasizes tool manipulation over underlying perceptual capability, leaving the model's visual encoder largely unchanged. There is no scalable, self-contained method that co-evolves visual and linguistic understanding.

Method

ViPER structures perceptual learning as a two-stage task and integrates data synthesis with RL into a closed loop.

Stage 1 — Caption Self-Refining: The VLM generates a caption \(C_g\) for image \(I\). A diffusion model (Qwen-Image) reconstructs an image from \(C_g\); inherent information loss creates local visual discrepancies. The VLM then predicts a set of refinement actions \(R_\text{pred} = f(I, C_g; \theta)\) that correct errors in object attributes, text, and spatial relationships. Objective: \(\min_\theta \mathbb{E}[\delta(R, R_\text{pred})]\).

Stage 2 — Visual-Operation Predicting: Using the refined caption, the VLM selects hard entities and generates visual editing instructions via heuristic rules. A second diffusion model (OmniGen2) executes these instructions to produce an edited image \(I_\text{recon}\). The VLM then predicts the applied operations \(\text{Ops}_\text{pred} = f(I_\text{orig}, I_\text{recon}; \theta)\) from the image pair. The instructions serve as self-generated ground truth.

Reward: A composite reward \(R = w_f R_\text{format} + w_c R_\text{correct}\) (\(w_f{=}0.05\), \(w_c{=}0.95\)) where \(R_\text{correct}\) uses BGE-M3 semantic similarity with threshold \(\tau{=}0.85\), weighted by sentence length.

Optimization: A DAPO-modified GRPO variant with decoupled clipping bounds \((\epsilon_\text{low}, \epsilon_\text{high})\) and no KL penalty, applied sequentially across the two stages on the Viper10K dataset (7K stage-1 + 3K stage-2 samples). The VLM within the synthesis module is updated from training checkpoints, enabling co-evolution of model and training data.

Key Contributions

  • Two-stage coarse-to-fine task formulation that separates static holistic understanding (caption self-refinement) from dynamic fine-grained reasoning (visual-operation prediction).
  • ViPER self-evolutionary closed-loop framework: VLM generates training data via diffusion-model reconstruction, which feeds directly into RL training — no external data or cold-start required.
  • Viper10K dataset and the Qwen-Viper series of perception-enhanced models.
  • Empirical evidence for the reciprocal relationship between visual generation and understanding: enhanced generation capacity directly improves perceptual discrimination.
  • Demonstration that two-stage sequential RL outperforms mixed-data RL for coarse-to-fine skill acquisition.

Results

  • Average across 7 benchmarks: +1.7% (Qwen-Viper-3B vs. Qwen2.5-VL-3B), +1.6% (Qwen-Viper-7B vs. Qwen2.5-VL-7B).
  • Fine-grained Perception (MMStar subdomain): +4.4% (3B), +6.0% (7B) — largest gains.
  • Coarse Perception: +1.2% (3B), +1.6% (7B).
  • Instance Reasoning: +2.0% (3B), +1.6% (7B).
  • Science & Technology (7B): +3.6%, despite no domain-specific training.
  • HallusionBench: Reduced hallucination rates for both model sizes.
  • MANTIS Eval (multi-image): +1.3% (3B), +0.5% (7B), confirming out-of-domain generalization.
  • SFT baseline (Viper10K SFT on 7B): 64.6 overall vs. 66.0 for RL-trained Qwen-Viper-7B, showing RL superiority over SFT on the same data.
  • Cold-start ablation: Training without cold-start converges to marginally higher final reward than SFT cold-start + RL, eliminating reliance on external annotation (Gemini-2.5-Pro generated 1K CoT data tested).
  • Two-stage vs. mixed RL: Two-stage RL outperforms mixed RL across all six evaluated subdomains.

Limitations

  • Evaluated exclusively on the Qwen2.5-VL model family (3B and 7B); transferability to other VLM architectures (e.g., InternVL, LLaVA) is not demonstrated.
  • Viper10K is small (10K samples), raising questions about scaling behavior.
  • Data synthesis quality is bounded by the diffusion models used (Qwen-Image, OmniGen2); artifacts or failures in image reconstruction propagate into RL training signals.
  • Absolute gains on most benchmarks are modest (~1–2%), limiting significance for tasks that do not demand fine-grained perception.
  • The heuristic rules for generating visual editing instructions (Stage 2) are not fully detailed in the main text, reducing reproducibility clarity.
  • The paper text is truncated, so full ablation results (attention visualization, cross-stage interaction analysis) are not assessed here.

Relevance to Vision-Language Models

ViPER directly targets a persistent bottleneck in VLM post-training: reinforcement learning methods that reward text-level reasoning do not improve the model's underlying visual encoder. By constructing a closed-loop pipeline where the VLM's own generative outputs (mediated by diffusion reconstruction) serve as RL training signals, ViPER offers a scalable, annotation-free alternative to distillation-based data pipelines — relevant to any VLM training recipe that must improve perception without access to stronger teacher models. The demonstrated reciprocal generation–understanding dynamic connects to emerging research on multimodal self-improvement and suggests that training VLMs to generate (or critique generated) visual content is a viable path to stronger visual grounding, complementing the dominant text-reasoning RL paradigm seen in Vision-R1 and related work.