Skip to content

Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Tar introduces a unified multimodal LLM that handles both visual understanding and generation through a single shared discrete semantic representation. The core innovation, Text-Aligned Tokenizer (TA-Tok), initializes a VQ codebook from frozen LLM token embeddings, projecting vision into text's latent space without modality-specific adapters. Tar matches or surpasses specialized continuous-token models on standard VQA and text-to-image benchmarks with faster convergence.

Problem

Existing unified MLLMs either use separate visual encoders for understanding vs. generation (Janus), pixel-level discrete tokens that conflict with high-level semantics (Chameleon, Emu3), or hybrid tokenizers that fail to converge optimally for both tasks (UniTok, VILA-U). No prior work achieves a fully discrete, semantically aligned, shared representation that unifies both tasks without modality-specific components or separate alignment stages.

Method

Text-Aligned Tokenizer (TA-Tok): Images are encoded by a frozen SigLIP2 encoder, then quantized via vector quantization against a codebook initialized from top-k LLM token embeddings (Qwen2.5, 65536 entries). Only a projection matrix W is learned; codebook entries are C = EW (E frozen), ensuring visual tokens live in the LLM's latent space. A lightweight ViT decoder (SAD) reconstructs SigLIP2 teacher features via cosine reconstruction loss, guiding semantic fidelity.

Scale-Adaptive Pooling/Decoding (SAP/SAD): A single scale factor s ∈ {1, 2, 3} controls token count (729/169/81 tokens at 384px), enabling fine-grained understanding or coarse generation at inference.

Generative De-Tokenizers: Two complementary decoders: (1) AR-DTok — an autoregressive LLaMA-style model predicting VQVAE tokens conditioned on TA-Tok outputs; (2) Dif-DTok — pretrained SANA-0.6B diffusion model finetuned with TA-Tok tokens as cross-attention conditioning (only cross-attention and condition embedding layers updated).

Unified MLLM (Tar): Qwen2.5-Instruct backbone with expanded embedding matrix {E, EW} covering both text and visual tokens. Trained with standard cross-entropy on mixed sequences. Novel pretraining tasks added: image-to-image (I→I, two FLUX images from same prompt) and text-image-to-image (TI→I, reference image + partial prompt → target image), improving modality alignment.

Key Contributions

  • TA-Tok: Text-aligned VQ codebook initialized from frozen LLM embeddings, eliminating a separate visual-language alignment stage.
  • Scale-Adaptive Pooling/Decoding: Multi-granularity visual tokens from a single tokenizer, adapting to understanding (fine) vs. generation (coarse) needs.
  • Generative De-Tokenizers: AR and diffusion variants that decode high-fidelity images from semantic discrete tokens.
  • Novel pretraining tasks (I→I, TI→I) that bridge understanding and generation, improving generation by +4.3 DPG points over baseline.
  • Self Reflect strategy: Tar uses its own VQA capability to score image-prompt alignment and resample, boosting GenEval from 0.76→0.84 (7B) and DPG from 84.19→84.65.

Results

Visual Understanding (Tab. 1): - Tar-1.5B: POPE 88.4, MME-P 1390, GQA 61.1, MMMU 38.2 — surpasses most understanding-only and unified models at same scale. - Tar-7B: matches Janus-Pro-7B (state-of-the-art continuous-token model) on MMMU (41.8 vs. 41.0), POPE (88.5 vs. 87.8), GQA (62.0 vs. 61.3).

Visual Generation (Tab. 2): - Tar-1.5B + Self Reflect: GenEval 0.78, DPG 84.19 — outperforms Janus-Pro-1B (0.73 / 82.63) and approaches Janus-Pro-7B (0.80 / 84.19). - Tar-7B + Self Reflect: GenEval 0.85, DPG 84.65 — surpasses all unified models including Janus-Pro-7B.

Tokenizer comparison (Fig. 5, controlled 10M data): - TA-Tok: Understanding 92–93, Generation DPG 70.6 — best balance; VQVAE ~74/48; Janus ~94/33; Hybrid ~73/65.

Joint training benefit (Tab. 6): - TA-Tok gains +5.3 DPG on generation from joint training; VQVAE gains +8.1; Janus loses −0.1 (no shared representation).

Limitations

  • AR-DTok is trained at 256px resolution from scratch, limiting generation resolution; higher resolutions require costly additional finetuning stages.
  • Self Reflect strategy (resampling based on model's own alignment score) is not analyzed for failure modes or compute overhead.
  • Scale-Adaptive Pooling with 729 tokens for understanding creates long sequences that slow LLM training; 169 tokens are used as a compromise.
  • Text-aligned codebook selects top-k LLM embeddings by inter-embedding distance heuristic — may not optimally cover visual semantic space.
  • Evaluation is limited to standard 2D benchmarks; no video, 3D, or multi-turn editing tasks evaluated despite motivation.

Relevance to Vision-Language Models

This work directly advances the fundamental question of how to represent images in a shared space with language tokens, a core bottleneck in building truly unified MLLMs. By grounding visual quantization in LLM token embeddings rather than pixel reconstruction, TA-Tok provides a cleaner pathway to native multimodal generation without the modality-gap bridging that plagues methods like LLaVA-style adapters or Janus-style dual encoders. The result that a fully discrete semantic representation matches continuous-token models on VQA challenges the prevailing assumption that discretization is too lossy for high-quality understanding. For researchers tracking VLMs, this paper offers concrete design principles — LLM-anchored codebook initialization, scale-adaptive tokenization, and cross-modal pretraining tasks — that are transferable to other unified architectures.