Skip to content

UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokens

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

UniCTokens is a personalization framework for unified vision-language models that replaces separate task-specific concept tokens with a single set of shared tokens trained via a three-stage progressive strategy. It demonstrates that understanding and generation mutually reinforce each other, enabling a novel task called personalized attribute-reasoning generation—producing images that reflect textual attributes not visible in training images. It achieves state-of-the-art results on the new UnifyBench benchmark at 1.3B parameters with only 3–10 training images per concept.

Problem

Existing personalized VLMs and diffusion models treat understanding and generation as independent tasks with separate concept tokens, failing to exploit their complementary semantics. This isolation prevents cross-task information transfer: a diffusion model cannot generate "⟨concept⟩ wearing its hat" if the hat's color was only described in the understanding corpus. No benchmark existed to jointly evaluate understanding, generation, and attribute-reasoning generation in a unified model.

Method

UniCTokens fine-tunes a set of unified concept tokens on the Show-o (1.3B) autoregressive VLM backbone using three progressive stages:

  1. Stage 1 – Understanding Warm-up: Soft-prompt tuning with shared tokens ⟨tokens_s⟩ (K=16) via VQA instruction tuning (recognition, random, conversational). User text descriptions are converted to text-only QA pairs and included. Token initialization follows MC-LLaVA.

  2. Stage 2 – Bootstrap Generation from Understanding: Adds generation-specific tokens ⟨tokens_g⟩ (M=8). Shared tokens transfer understanding priors to T2I training. For human concepts, a facial encoder initializes ⟨tokens_g⟩; for non-human, Stage 1 strategy is reused. Only 3–10 samples suffice because understanding priors bootstrap generation.

  3. Stage 3 – Deepen Understanding from Generation: Exploits Show-o's MaskGIT-style decoding: intermediate image differences (k-means clustering on xm − xn across timesteps) identify "hard regions" encoding fine-grained concept details. These regions initialize new understanding tokens ⟨tokens_u⟩ (N=8). Stage 3 only updates ⟨sks⟩ and ⟨tokens_u⟩ via recognition VQA, injecting generative perceptual priors back into understanding.

Final concept prompt: ⟨sks⟩ is ⟨tokens_s⟩ ⟨tokens_u⟩; generation prompt: A photo of ⟨tokens_s⟩ ⟨tokens_g⟩ ⟨sks⟩.

Key Contributions

  • Unified concept tokens: Single shared token set for both understanding and generation, replacing the dual-token paradigm of Yo'Chameleon.
  • Three-stage progressive training: Structured cross-task information transfer (warm-up → bootstrap generation → deepen understanding).
  • Generation-as-perception: Using intermediate diffusion timestep differences to localize concept-critical regions for understanding token initialization.
  • UnifyBench: First benchmark jointly evaluating concept understanding, concept generation, and the new personalized attribute-reasoning generation (PARG) task across 20 concepts (persons, pets, objects).

Results

  • UnifyBench understanding: UniCTokens improves vanilla Show-o by avg. 8.9%; achieves Rec=0.790, VQA=0.503, QA=0.544 with only ∼10 training images vs. Yo'Chameleon's ∼1,000.
  • UnifyBench generation (pure gen.): CLIP-I=0.750, CLIP-T=0.282, DINO=0.646 — best among unified models; People Gen. Face-Simi=0.334 vs. Yo'Chameleon's 0.224.
  • PARG (attribute-reasoning generation): PARG Score=0.359 vs. Yo'Chameleon's 0.266 and Show-o+TP's 0.770-but-low-CLIP-I; UniCTokens best balances PARG score and CLIP-I (0.749).
  • External benchmarks (Yo'LLaVA dataset): Rec=0.852, VQA=0.615, QA=0.738 — outperforms Yo'Chameleon (0.845/0.604/0.721) and all 1.3B-scale models.
  • DreamBench generation: CLIP-I=0.800, CLIP-T=0.287 — surpasses Yo'Chameleon (0.795/0.225) and DreamBooth (0.803/0.271) with only 10 training images vs. DreamBooth's 3,000.
  • Ablation: Removing Stage 1 causes large PARG degradation (Score 0.334→0.204); Stage 3 boosts Rec from 0.621 to 0.790 without degrading generation.

Limitations

  • Built on Show-o 512×512; generation resolution is constrained and backbone-dependent.
  • UnifyBench is small (20 concepts), limiting statistical generalization of benchmark conclusions.
  • Multi-concept personalization is not evaluated; the paper explicitly restricts to single-concept MC-LLaVA tests due to the base model's limitations.
  • Stage 3 token initialization relies on MaskGIT-style decoding; the method is not directly applicable to diffusion-only or AR-only backbones without adaptation.
  • Training three sequential stages per concept increases total fine-tuning complexity compared to single-stage methods.

Relevance to Vision-Language Models

UniCTokens directly addresses the fragmentation between VLMs (understanding) and diffusion models (generation) by personalizing a single autoregressive VLM for both tasks, advancing the line of unified multimodal models (Chameleon, Janus, Show-o). The finding that understanding quality causally improves generation—and that generation intermediate states can recover fine-grained understanding features—is a concrete architectural insight for building personalized VLMs. The PARG task formalizes cross-modal knowledge transfer as a measurable capability, providing a new evaluation axis for the community. UnifyBench fills a gap in the absence of any joint understanding+generation+reasoning benchmark for personalized VLMs.