Skip to content

Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLV (Vision-Language-Vision) Auto-Encoder distills semantic knowledge from a frozen Stable Diffusion 2.1 decoder into a compact continuous embedding space using only single-modal images, then decodes those embeddings into captions via a fine-tuned LLM. The result is a captioner competitive with GPT-4o and Gemini 2.0 Flash trained for under $1,000 USD (< 1,000 GPU hours). The framework also exhibits emergent spatial awareness and compositional generalization.

Problem

State-of-the-art VLM captioners require billions of high-quality image-text pairs and millions of GPU hours. Existing methods for learning language-aligned visual representations either depend on massive paired datasets, rely on discrete tokenization (Gumbel-Softmax) that loses fine-grained semantics, or require co-training of text and vision modules from scratch.

Method

Stage 1 — VLV Auto-Encoding (image-only, self-supervised): A Florence-2-initialized image encoder produces visual tokens, which a lightweight Transformer encoder-decoder (with Nq=77 learnable cross-attention queries) maps into continuous caption embeddings z ∈ ℝ^(77×d_CLIP). These embeddings are fed as conditioning to a frozen Stable Diffusion 2.1 U-Net, which reconstructs the image via the standard denoising loss L_denoise. Because the diffusion decoder is frozen, the encoder is forced to compress all reconstruction-relevant information into z, effectively distilling the T2I model's cross-modal knowledge without any text supervision.

Stage 2 — Caption Decoding (paired image-text, 6M samples): z is passed through the frozen CLIP text encoder to obtain contextual representations c, projected by a trainable MLP ψ to the LLM's hidden dimension, then prepended to token embeddings in a causal Qwen-2.5 LM. The autoregressive cross-entropy loss trains only {VLV encoder, MLP ψ, LLM decoder}; the CLIP encoder and diffusion decoder remain frozen. At inference, z → c → e → autoregressive caption, with no text tokens as input.

Key Contributions

  • VLV Auto-Encoder framework: uses a frozen T2I diffusion decoder as an information bottleneck to learn language-semantic image representations from unpaired images only.
  • Two-stage pipeline that separates semantic distillation (image-only) from caption decoding (small paired dataset), decoupling data requirements across stages.
  • Total training cost under $1,000 USD on 8× RTX 6000 Ada GPUs (~4 days Stage 1, shorter Stage 2).
  • Demonstrated emergent properties: 6-DoF spatial awareness and zero-shot compositional generalization via embedding concatenation.

Results

  • FID (caption→SD 3.5 reconstruction, MS-COCO 2014, 30K samples): VLV achieves FID 6.64 at guidance scale 2.0 vs. GPT-4o 6.20 and Gemini 2.0 Flash 5.87 (best); outperforms Florence-2 Large (7.51) and Qwen2.5-VL-7B (6.98).
  • Human + VLM caption rating (0–6 scale): VLV 5.18 vs. GPT-4o 5.23 vs. Qwen2.5-VL-7B 5.03; gap to GPT-4o < 0.05 points.
  • VQA (32-shot, text-only via DeepSeek-V3): VQAv2 63.60% vs. Qwen2.5-VL-7B 63.77% (best open-source); OK-VQA 60.25% vs. Gemini 60.25%; within 1 point of SOTA at 32-shot.
  • Scalability: FID improves monotonically from 6M→18M→40M training images; decoder scaling from 0.5B→3B Qwen-2.5 similarly improves FID (14.70→11.47 at guidance 1.0).
  • Spatial awareness (L1 bounding-box deviation): Angle error drops from 0.1564 (6M images) to 0.1016 (40M images); center and scale errors show comparable improvements.

Limitations

  • Poor OCR performance due to aesthetic-score filtering that excludes text-heavy images.
  • Upper bound constrained by Stable Diffusion 2.1 (outdated decoder); re-distillation from SD 3.5 or FLUX is noted as future work.
  • 6M paired image-text samples for Stage 2 are synthetic (Gemini 2.0 Flash generated), introducing dependence on a proprietary model for alignment data.
  • Video extension is unexplored; dynamics and temporal consistency are not addressed.
  • Zero-shot VQA accuracy trails SOTA by ~3 percentage points, indicating some loss of fine-grained grounding relative to fully supervised VLMs.

Relevance to Vision-Language Models

VLV directly challenges the assumption that high-quality captioning requires massive paired image-text pretraining, showing instead that a frozen generative decoder can serve as a sufficient supervisory signal to learn language-compatible visual representations. For the VLM community, this opens a pathway to training competitive captioners at negligible cost by exploiting pretrained T2I models as knowledge sources rather than treating them as orthogonal to discriminative/generative VLMs. The emergent compositionality and spatial 3D awareness in the learned embeddings suggest that diffusion-distilled representations encode richer structure than standard CLIP embeddings, which is relevant for downstream grounding and spatial-reasoning tasks in VLMs. The framework also exemplifies a modular, frozen-component design philosophy (encoder + frozen diffusion decoder + LLM) that aligns with efficient VLM adaptation trends such as BLIP-2 and LLaVA.