Skip to content

Selftok-Zero: Reinforcement Learning for Visual Generation via Discrete and Autoregressive Visual Tokens

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Selftok-Zero applies reinforcement learning (RL) post-training to a pure autoregressive (AR) vision-language model for text-to-image generation, enabled by the Selftok tokenizer which encodes images as causally-ordered 1D discrete tokens derived from the reverse diffusion trajectory. Without any paired text-image supervision during RL, it achieves 92% on GenEval and 85.57 on DPG-Bench, surpassing all compared baselines. The core argument is that only true AR tokens satisfy the Bellman equation required for valid policy improvement in RL.

Problem

Two fundamental incompatibilities block RL post-training for visual generative models: (1) diffusion-based models face intractable diffusion inversion, causing a large mismatch between the behavior policy (linear forward diffusion) and the target policy (non-linear reverse trajectory), yielding poor action-space coverage; (2) standard spatial visual tokens in AR models carry anti-causal dependencies—a later token's policy update can retroactively perturb earlier tokens—which violates the Bellman equation and the policy improvement theorem. Both failures prevent reliable reward signal propagation.

Method

Selftok tokenizer: Encodes image \(I\) into \(K\) discrete tokens \(V_K = [v_1, \ldots, v_K]\) that explicitly satisfy the AR causal graph. The key insight is to align the AR recursive structure with the ODE decomposition of the reverse diffusion process: given a noisy midpoint \(x_t \sim q(x_t|x_1)\), the encoder learns tokens \(V_{\geq k(t)}\) needed to reconstruct \(x_1\) from \(x_t\), with a token schedule \(k(t)\) mapping continuous diffusion time to discrete token indices. This grounds the tokens in the diffusion generation trajectory rather than spatial image patches, making them non-spatial and causally ordered.

VLM pre-training: A Llama3-8B model is extended with \(|\mathcal{C}|=32{,}768\) Selftok visual tokens and pre-trained on interleaved text-image data via standard next-token prediction.

RL (Selftok-Zero): A simplified GRPO without importance sampling. For each prompt, \(B\) token sequences are sampled; advantages are normalized per-batch. The loss is \(\mathcal{L} = -\frac{1}{B}\sum_i [A_i - \lambda D_{\text{KL}}(\pi \| \pi_{\text{old}})]\). Two reward types are used: (1) program-based rewards using MM-Detection (confidence-threshold object counting for structured prompts), and (2) QA-based rewards decomposing prompts into semantic tuples and querying InternVL/mPLUG as VQA judges. No pairwise text-image supervision is used during RL.

Key Contributions

  • Formal proof that spatial visual tokens violate the Bellman equation due to anti-causal collider effects, disqualifying them from theoretically sound RL optimization.
  • Selftok: a tokenizer grounded in reverse-diffusion trajectory recursion that enforces strict AR causality, enabling valid policy improvement.
  • Selftok-Zero: first demonstration of substantial RL-driven gains for open-vocabulary AR visual generation without any pairwise supervision.
  • Two complementary reward paradigms (program-based and QA-based) for evaluating generated image quality under RL.

Results

  • GenEval: Selftok-Zero overall 92% vs. Selftok-SFT 74% (+18), HiDream-I1 83%, CogView4-6B 73%, Janus-Pro-7B 80%, Janus-Pro-7B-Zero 85%.
  • Position: 45→96 (+51); Two Objects: 79→95 (+16); Counting: 66→88 (+22).
  • DPG-Bench: Selftok-Zero 85.57 vs. Selftok-SFT 81.80 (+3.77), SD3-Medium 84.08, Janus-Pro-7B 84.19, Janus-Pro-7B-Zero 84.49.
  • Entity: 88.15→91.78; Relation: 93.68→95.26.
  • Selftok-Zero (+18 GenEval) significantly exceeds Janus-Pro-7B-Zero (+6 GenEval) when both undergo the same RL process, isolating the tokenizer's AR property as the differentiating factor.
  • Program-based rewards yield larger GenEval gains than QA-based rewards yield on DPG-Bench, attributed to cleaner, more reliable reward signals.

Limitations

  • Image generation is restricted to 256×256 resolution, substantially limiting perceptual quality compared to state-of-the-art diffusion models.
  • Reward models are only preliminary (program-based detection + generalist VQA); more task-specific or fine-tuned reward models are left for future work.
  • Token generation speed is not optimized; no spatial-temporal compression is applied to Selftok sequences.
  • RL is validated on text-to-image; extension to high-resolution generation and physics-aware visual reasoning remains future work.
  • The simplified GRPO variant omits importance sampling, which may limit off-policy efficiency.

Relevance to Vision-Language Models

Selftok-Zero demonstrates a principled pathway to apply RL—already transformative for LLMs via methods like GRPO and DeepSeek-R1—to unified AR VLMs that handle both language and visual generation in a shared token space. The paper formalizes why existing VLM architectures using spatial patch tokens are theoretically incompatible with RL's policy improvement guarantees, which has direct implications for any VLM research involving preference optimization or reward-guided generation. For researchers tracking VLMs, this work repositions the image tokenizer design as a first-order concern for RL-based alignment, not merely a perceptual quality concern. It also establishes a concrete benchmark trajectory (GenEval 92%, DPG-Bench 85.57 without supervision) that future unified understanding-and-generation VLMs will need to contend with.