Skip to content

LMFusion: Adapting Pretrained Language Models for Multimodal Generation

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LMFusion adapts pretrained text-only LLMs (Llama-3 8B) for unified multimodal generation—both image understanding and diffusion-based image synthesis—by adding parallel, modality-specific transformer modules while freezing the original text weights. This avoids catastrophic forgetting and eliminates the need to retrain on text data. Compared to training from scratch (Transfusion), it achieves +20% image understanding and +3.6% image generation while fully preserving Llama-3's language scores.

Problem

Training unified multimodal generative models from scratch (Transfusion, Chameleon, Unified-IO) requires enormous compute—Llama-3 alone consumed 15T tokens. Naively fine-tuning a pretrained LLM on multimodal data with shared weights causes catastrophic forgetting: HellaSwag drops ~15% initially and never fully recovers, creating an inherent text-vision trade-off.

Method

LMFusion inserts a parallel set of image-specific transformer modules alongside each frozen Llama-3 layer. Within each transformer block: - Modality-specific QKV projections and FFNs: text tokens route through Llama-3's original QKV/FFN weights (frozen, η_text = 0); image patches (VAE-encoded, 32×32×8, compressed by a U-Net downsampler to 256 tokens) route through newly initialized image QKV/FFN weights. - Shared causal self-attention across modalities: queries, keys, and values from both modalities are concatenated before softmax, enabling cross-modal attention with a hybrid mask (causal for text, bidirectional for images). - Decoupled output projections: text hidden states decode to logits via LM head; image hidden states decode to noise predictions via U-Net upsampler. - Dual training objective: LM cross-entropy loss on text + DDPM loss (λ-weighted) on images, matching the Transfusion recipe. - No text data required: because text modules are frozen, text-only training tokens can be excluded entirely, halving effective training FLOPs vs. Transfusion.

The architecture is initialized with Llama-3 8B weights for both text and image modules; only image-specific parameters (QKV_img, FFN_img, U-Nets) are trained with AdamW (lr=1e-4), on 380M Shutterstock image-caption pairs over 4 days on 128 H100s.

Key Contributions

  • LMFusion framework: deep modality separation (modality-specific QKV + FFN + separate O projections) that enables reuse of pretrained LLM weights for multimodal generation without text degradation.
  • Controlled comparison: same data, same FLOP budget as Transfusion, demonstrating clear gains from the architecture choice rather than scale.
  • Ablation hierarchy: empirically ranks no separation < shallow separation (FFN only) < deep separation (FFN + attention), justifying full architectural decoupling.
  • LLaVAFusion: extends the recipe to an existing VLM (LLaVA-NeXT 8B), adding image generation ability while preserving multimodal understanding, outperforming EMU-3, Show-O, Janus, Chameleon, and MetaMorph on MMMU, ChartQA, RealWorldQA, and MME-P.

Results

  • Language (HellaSwag / SIQA / WinoGrande): LMFusion (both 0.5Ă— and 1.0Ă— FLOPs) scores 60.0 / 48.1 / 72.8 vs. Transfusion 51.0 / 42.3 / 64.3 — matching Llama-3 exactly, +11.6% over Transfusion.
  • Image Understanding (CIDEr on MS-COCO): LMFusion 38.3–38.4 vs. Transfusion 32.0 (+20%).
  • Image Generation (FID / CLIP on MS-COCO, without CFG): LMFusion FID 13.9–14.0 vs. Transfusion 14.4; CLIP 22.0–22.1 vs. 22.1 (marginal but consistent improvement; +3.6% overall generation quality cited).
  • LLaVAFusion vs. baselines: MMMU 41.7 (vs. EMU-3 31.6, MetaMorph 41.8); ChartQA 69.5 (vs. MetaMorph 37.1); RealWorldQA 60.0 (vs. MetaMorph 58.3); MME-P 1603.7 (vs. Janus 1338.0); FID 8.2 (best in table, vs. Show-O 9.2, Janus 8.5).
  • Ablation: with frozen text modules, deep separation > shallow separation > no separation on both CIDEr and FID throughout training.

Limitations

  • Image-specific modules are trained from scratch; only language weights are reused, so substantial compute is still required for the vision side.
  • Evaluation is limited to 256Ă—256 resolution images using a fixed VAE tokenizer.
  • The training dataset is exclusively Shutterstock image-caption pairs (380M); no instruction tuning or diverse multimodal corpora explored in main experiments.
  • The 2Ă— parameter count of LMFusion vs. Transfusion (even though training FLOPs match) increases inference cost.
  • Concurrent work (Mixture-of-Transformers, Playground v3) is not directly benchmarked against in the controlled setting.

Relevance to Vision-Language Models

LMFusion directly addresses a key challenge in extending VLMs to generative tasks: how to add image synthesis capability to a pretrained VLM without destroying its understanding capabilities. The modality separation principle—parallel transformer modules with shared attention but independent QKV/FFN—is a concrete architectural recipe that VLM practitioners can apply when adapting existing models like LLaVA. The LLaVAFusion extension shows this is not limited to text-only LLMs, making it immediately relevant to anyone building on top of instruction-tuned multimodal models. The result that deep (QKV+FFN) separation outperforms shallow (FFN-only) separation challenges the common MoE-style assumption that routing at the FFN level is sufficient for modality specialization.