Skip to content

Modality-Specialized Synergizers for Interleaved Vision-Language Generalists

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MoSS (Modality-Specialized Synergizers) is a parameter-efficient fine-tuning framework for Vision-Language Generalists (VLGs) that uses architecturally distinct LoRA adapters per modality: Convolutional LoRA for image patches (to capture local spatial priors) and Linear LoRA for text tokens. Paired with LeafInstruct, the first open-sourced large-scale interleaved instruction-tuning dataset (184,982 instances), MoSS achieves state-of-the-art interleaved text-and-image generation among open-source models.

Problem

Existing VLGs apply a unified transformer architecture with shared parameters to both discrete text tokens and continuous image features, ignoring their fundamentally different inductive biases—text is sequential while images have 2D local spatial structure. Prior modality-aware expert approaches (e.g., MoE-LoRA) still use identical linear architectures for both modalities. Additionally, no large-scale instruction-tuning data existed for interleaved (mixed text+image) output generation; existing datasets only cover single-modality outputs.

Method

MoSS injects two modality-specialized lightweight adaptation layers into each linear layer of a frozen VLG backbone:

  • Linear LoRA: Standard low-rank decomposition applied to hidden states corresponding to text token positions.
  • Convolutional LoRA: A novel variant where image hidden states are reshaped into 2D spatial grids (H×W×C), passed through a causal Conv2D kernel (top/left zero-padded to respect autoregressive ordering), then projected back via a linear LoRA B matrix. Convolution operates in the original feature dimension (not a reduced space), avoiding information loss from premature dimensionality reduction.

During fine-tuning, the VLG backbone is frozen; only the two LoRA sets are updated. Hidden states are routed by output modality: text-generating positions → Linear LoRA, image-generating positions → Convolutional LoRA.

LeafInstruct is constructed from web/academic sources (MMDialog, VIST, WikiWeb2M, YouCook2) via an automatic annotation pipeline, filtering 184,982 high-quality interleaved instances from >7M source samples across 10+ domains.

Key Contributions

  • MoSS framework: First PEFT method applying distinct LoRA architectures (linear vs. convolutional) to text and image modalities within a single autoregressive VLG.
  • Convolutional LoRA: Causal 2D convolutional adaptation that models local patch priors during autoregressive image generation; convolution performed at full input dimension before rank reduction.
  • LeafInstruct: First open-sourced, large-scale interleaved instruction-tuning dataset (184,982 instances, 10+ domains) with interleaved text+image inputs and outputs.
  • Demonstrated generalizability across two architecturally different VLG backbones (Emu2: continuous image tokens; Chameleon: discrete image tokens).

Results

Evaluated on InterleavedBench using InterleavedEval (GPT-4o scoring 0–5 on five aspects):

  • Emu2 + MoSS vs. Emu2 baseline: +96.2% Text Quality, +58.1% Image Coherence, +99.4% Text-Image Coherence (TIC), +190.2% TIC, +44.9% Helpfulness; ~+97.8% average across 5 aspects.
  • Emu2 + MoSS vs. best open-source baseline (GILL): +34.7% overall average, +37.8% Image Coherence, +31.6% TIC, +11.5% Helpfulness.
  • MoSS vs. MoE-LoRA (same modality routing, linear architecture for both): MoSS strictly better across all five aspects, demonstrating that architectural specialization (conv vs. linear) matters beyond just parameter separation.
  • MoSS performance gap vs. proprietary pipelines (GPT-4o+DALL·E 3, Gemini1.5+SDXL): gap remains, especially on Perceptual Quality and TIC.
  • Rank ablation: MoSS consistently outperforms LoRA and MoE-LoRA at all ranks (32, 64, 128, 256); gap widens with larger rank.
  • LeafInstruct human evaluation (200 sampled instances, 0–3 scale): near-ceiling scores across all five aspects (Text Quality 2.89, Perceptual Quality 2.96, Image Coherence 2.77, TIC 2.87, Helpfulness 2.71).

Limitations

  • Slight Text Quality degradation when applying MoSS to Chameleon, attributed to trade-offs in interleaved instruction tuning.
  • Perceptual quality of generated images remains below proprietary pipelines (GPT-4o+DALL·E 3); complex tasks like image editing and subject-driven generation still produce noticeable distortions.
  • Rank 256 is required to realize MoSS's full advantage, increasing parameter count relative to lower-rank baselines.
  • Convolutional LoRA introduces non-trivial integration complexity into autoregressive decoding (causal padding, 2D reshape/flatten per image sequence).
  • LeafInstruct construction relies on an automated pipeline; domain coverage is bounded by the four source datasets.

Relevance to Vision-Language Models

MoSS directly addresses a fundamental architectural mismatch in VLGs—the tension between transformer's sequential inductive bias and the spatial locality of images—by grounding modality-specific adaptation in distinct inductive priors rather than just separate parameter sets. This is a principled extension of PEFT to the multimodal generation setting and informs how future unified VLMs should allocate capacity across modalities. LeafInstruct fills a critical data gap for interleaved generation, enabling reproducible instruction tuning for the community. The work is particularly relevant for researchers studying mixed-modal generation, efficient adaptation of large VLMs, and the role of architectural inductive biases in cross-modal learning.