Skip to content

Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Bifrost-1 bridges pretrained multimodal LLMs (MLLMs) and diffusion models using 2D patch-level CLIP image embeddings as latent variables, exploiting the native alignment between the MLLM's own CLIP visual encoder and those embeddings. A visual generation branch (initialized from frozen MLLM weights) predicts masked patch embeddings autoregressively; a lightweight latent ControlNet then injects these into FLUX.1-dev for final image synthesis. The design achieves competitive understanding and generation quality with substantially lower training compute than prior single-architecture or 1D-bridging methods.

Problem

Existing unified multimodal generation methods either (a) fine-tune the entire LLM backbone for image generation—expensive and prone to reasoning degradation—or (b) bridge LLMs and diffusion models via sparse text descriptions or 1D continuous query tokens that require large connectors and end-to-end training, losing fine-grained 2D spatial detail. Neither paradigm efficiently leverages the existing alignment between pretrained MLLMs and CLIP embeddings.

Method

Bifrost-1 adds a visual generation branch to a frozen MLLM (Qwen2.5-VL 3B or 7B): it copies the MLLM's attention QKV projections, MLP, and normalization layers into a trainable parallel branch and appends a single linear vision head. During training, a random fraction (70–100%) of input patch CLIP embeddings are replaced with a learnable <M> token; the branch predicts the masked embeddings via MSE loss using masked autoregression (following MAR). At inference, generation starts from fully masked tokens and iteratively denoises them.

Predicted patch embeddings are spatially downsampled 2× via a lightweight 2D convolution and fed to a latent ControlNet—a partial copy of FLUX.1-dev (4 MM-DiT double-stream blocks + 1 single-stream block) whose input projection is replaced to accept MLLM-dimension CLIP features. Training is decoupled: the MLLM branch and latent ControlNet are trained separately to handle their different convergence rates and memory requirements. The MLLM backbone parameters remain fully frozen throughout.

Key Contributions

  • Patch-level CLIP latents as a 2D communicative bridge between MLLMs and diffusion models, natively aligned with the MLLM's own visual encoder—eliminating the realignment cost of prior methods.
  • Latent ControlNet: a lightweight ControlNet variant that accepts CLIP embeddings (not pixel-space images) to inject spatial structure into FLUX.1-dev.
  • Visual generation branch initialized from frozen MLLM parameters, so only one linear layer is randomly initialized, drastically reducing training cost.
  • Decoupled training strategy separating MLLM branch training from ControlNet training.
  • Full preservation of MLLM backbone parameters, retaining all original multimodal reasoning capabilities.

Results

  • ImageNet 256Ă—256 design comparison (Table 1): Bifrost-1 achieves FID 25.77 / sFID 53.67 / IS 98.57 vs. next-best variant (2D learnable query tokens): FID 118.69 / IS 9.15; vs. VAE latent variant: FID 284.51; vs. SigLIP variant: FID 274.16.
  • Multimodal understanding (Table 2, 7B): Matches Qwen2.5-VL 7B exactly (MME-P 1685.2, MMB 83.5, SEED 76.9, MMMU 58.6, MM-Vet 66.6), outperforming all baselines that fine-tune the backbone (e.g., JanusPro-7B: MMMU 50.6, MM-Vet 60.1).
  • Text-to-image generation (Table 3): Bifrost-1 7B (62M training images) achieves GenEval 0.81 (matching MetaQuery-XL, Transfusion, Janus), DPG-Bench 77.67 (outperforms TokenFlow-XL 73.38), COCO FID 34.35, MJHQ FID 16.21—with 62M vs. 100–211M training images for comparable methods.
  • Scaling tokens (Fig. 4): 256 CLIP latent tokens (14Ă—14) converge faster and reach better rFID/SSIM/PSNR/LPIPS than 16/64/144 tokens within 1 epoch (~26K steps).
  • Decoding steps (Table 4): Robust down to 8 steps (FID 18.89); degrades sharply below 4.

Limitations

  • COCO FID scores (23.02 / 34.35) are weaker than several baselines (e.g., MetaQuery-L 8.87), partly attributed to FLUX.1-dev's aesthetic fine-tuning skewing FID statistics.
  • Overall generation quality does not yet surpass state-of-the-art methods despite efficiency gains.
  • Diffusion backbone (FLUX.1-dev, 12B params) dominates inference time (14.79s vs. <5.21s for MLLM), making the system heavyweight at inference.
  • Training set limited to 9M–62M images; performance gap with methods trained on larger datasets remains open.
  • Evaluated only on static image generation; video and other modalities not explored.

Relevance to Vision-Language Models

Bifrost-1 directly addresses the core tension in VLM research between preserving strong language/visual reasoning and enabling high-fidelity image synthesis, showing that freezing the MLLM backbone entirely—and exploiting its native CLIP alignment—is both feasible and more compute-efficient than fine-tuning. The work demonstrates that patch-level 2D spatial representations are qualitatively superior to 1D query tokens or text descriptions as a bridge medium, which has implications for how VLMs should be architected for multimodal generation tasks. Its decoupled training strategy and lightweight ControlNet adaptation provide a practical blueprint for upgrading any CLIP-aligned MLLM (e.g., Qwen2.5-VL, LLaVA-style models) with generation capabilities at low cost. This positions CLIP encoders not just as perception backbones but as the natural latent space for unified understanding-generation systems.