Skip to content

Chain-of-Zoom: Extreme Super-Resolution via Scale Autoregression and Preference Alignment

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Chain-of-Zoom (CoZ) is a model-agnostic framework that extends any fixed-scale SR backbone to extreme magnifications (up to 256Ă—) by factorizing super-resolution into an autoregressive chain of intermediate scale-states. A VLM fine-tuned with GRPO-based RLHF generates multi-scale-aware text prompts at each zoom step, compensating for diminishing visual evidence at high magnifications. No retraining of the SR backbone is required.

Problem

State-of-the-art diffusion-based SR models are rigidly bound to their training scale factor (e.g., 4×) and collapse—producing blur, artifacts, or hallucinations—when pushed to higher magnifications. Training new models for extreme factors (16×, 32×, 256×) is computationally prohibitive, and existing autoregressive or inverse-problem methods do not scale to these regimes.

Method

CoZ introduces scale-level AR-2 modeling: the conditional distribution p(x_H | x_L) is decomposed into a Markov-2 chain of intermediate scale-states x_0, x_1, …, x_n, each up-sampled by a fixed factor s using the backbone SR model. At each step i, a VLM generates a multi-scale-aware text prompt c_i conditioned on both x_{i-1} and x_{i-2} (two previous scales) to suppress hallucinations caused by sparse visual conditioning. The SR backbone then upsamples x_{i-1} conditioned on c_i. The prompt-extraction VLM (Qwen2.5-VL-3B) is fine-tuned using GRPO with a composite reward: - R_critic: a stronger critic VLM (InternVL2.5-8B) scores prompt semantic quality on [0,1]; - R_phrase: penalizes multi-image viewpoint markers ("first image", "second image"); - R_rep: n-gram repetition penalty.

The backbone SR model (OSEDiff on Stable Diffusion 3.0) requires no modification; CoZ wraps it at inference time using sliding-window crops for full-image processing.

Key Contributions

  • Scale-level AR-2 framework that decomposes extreme SR into tractable sub-problems, enabling any SR backbone to reach 256Ă— without retraining.
  • Multi-scale-aware prompt extraction: VLM conditioned on two consecutive scale-states to reduce hallucinations at high magnifications.
  • First application of GRPO-based RLHF to VLM prompt extraction for super-resolution, with a critic VLM as the preference reward model.
  • Model-agnostic design validated on two backbones (OSEDiff, S3Diff).

Results

  • At 256Ă— on DIV2K, CoZ (VLM) achieves NIQE 9.83, MUSIQ 47.83, MANIQA 0.569, CLIPIQA 0.599 vs. direct SR (NIQE 16.17, MUSIQ 28.89, MANIQA 0.447, CLIPIQA 0.520).
  • At 64Ă— on DIV2K, CoZ (VLM) NIQE 8.23 vs. direct SR 16.59; MUSIQ 52.13 vs. 22.54.
  • S3Diff + CoZ at 256Ă—: MUSIQ 43.59 vs. S3Diff direct 25.92 on DIV2K; NIQE 10.77 vs. 16.78.
  • At low scales (4Ă—), differences between methods are minimal across all metrics.
  • GRPO-tuned VLM prompts outperform DAPE prompts at high scales; DAPE degrades comparably to direct SR at 256Ă—.
  • Runtime at 256Ă—: SR phase ~0.58s, VLM prompt extraction ~1.35s per image (single image, DIV2K average).

Limitations

  • Error accumulates over repeated zoom iterations; no explicit correction mechanism.
  • Inference requires multiple sequential CoZ applications for full-image extreme SR (windowed crop-and-resize approach).
  • No full-reference metrics available for 256Ă— evaluation (ground truth does not exist); relies solely on no-reference perceptual metrics.
  • High-fidelity reconstruction from low-resolution inputs raises misinformation and privacy risks.

Relevance to Vision-Language Models

CoZ demonstrates a concrete application of VLMs as semantic guidance modules within a vision pipeline: the VLM is not generating text for understanding but for steering a generative model, an increasingly important role. The GRPO-based RLHF pipeline for fine-tuning the prompt-extraction VLM is directly applicable to other VLM alignment tasks requiring non-differentiable, perception-based reward signals. The multi-scale visual conditioning strategy—feeding two consecutive image states to the VLM to reduce hallucination—offers a transferable design pattern for VLMs operating in iterative or hierarchical generation pipelines. This work extends the body of evidence that VLM text embeddings substantially constrain ill-posed vision inverse problems.