Text to Sketch Generation with Multi-Styles¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

M3S is a training-free framework for text-guided sketch generation that enables style transfer from one or more reference sketches using diffusion models. It avoids content leakage—a key failure mode of prior K/V injection methods—by blending reference and target features with linear smoothing rather than direct substitution. Multi-style generation is enabled via a joint AdaIN module that lets users interpolate between two reference styles with a scalar parameter η.

Problem¶

Existing sketch synthesis methods (e.g., DiffSketcher, CLIPasso) offer limited stylistic control: text prompts cannot precisely convey fine-grained stroke characteristics, and direct K/V matrix swapping in self-attention—used by prior style transfer work—causes content leakage and structural artifacts when reference and target domains are structurally dissimilar (as is typical with sparse, abstract sketches).

Method¶

M3S operates on a frozen Stable Diffusion (v1.5 or SDXL) backbone with four coordinated components:

Hybrid attention fusion with linear smoothing: Reference K/V features (extracted via null-text inversion) are concatenated with target K/V features. A linear blending coefficient λ mixes reference and target features before concatenation: K̂_ref = λK_tar + (1−λ)K_ref, reducing content leakage while preserving style.
Joint AdaIN modulation: For multi-style generation, latent noise at each denoising step is modulated as z̃_t = η·AdaIN(z_t, z_t^ref1) + (1−η)·AdaIN(z_t, z_t^ref2), where η ∈ [0,1] controls style tendency. Dense vs. sparse stroke statistics propagate through the mean/variance of the AdaIN operation.
Style-content guidance: A dual classifier-free guidance formulation separates content direction (text-conditioned, no feature injection) and style direction (null-text conditioned, with feature injection) via independent guidance scales ω₁ and ω₂.
Counter-based regulation: Sobel-operator edge loss applied to Tweedie-denoised latents suppresses chaotic strokes in abstract sketch synthesis (γ=60 default).

Key Contributions¶

Training-free, zero-shot multi-style sketch generation from text prompts and reference sketch exemplars.
Linear feature blending strategy that provably reduces content leakage compared to direct K/V substitution or AdaIN-constrained alignment (StyleAligned).
Joint AdaIN module for continuous interpolation between two reference styles via a single scalar η.
Separated style-content guidance directions decoupled from standard CFG.
Demonstrated on both SD v1.5 and SDXL backbones; code released at https://github.com/CMACH508/M3S.

Results¶

CLIP-T (text alignment): M3S (SDXL) achieves best average of 0.3514 across 6 style datasets, outperforming all baselines including InstantStyle (0.3480), StyleAligned (0.3095–0.3137), and CSGO (0.3232–0.3336).
DINO style consistency: Slightly below AttentionDistillation (e.g., 0.7027 vs. 0.6922 for Style 6), reflecting the deliberate style-content trade-off; adjustable via style-oriented parameters (SDXL∗).
Human preference: M3S (SDXL) scores 6.19/8, M3S (SD v1.5) scores 5.44/8, both above all baselines (next best: AttentionDistillation 4.28, InstantStyle 5.08) across 44 validated questionnaires.
Statistical significance: M3S (SDXL) significantly outperforms InstantStyle (p=1.06×10⁻⁵); M3S (SD v1.5) vs. InstantStyle is not significant (p=0.26).
Multi-style: Stable CLIP-T (~0.35) across η ∈ {0, 0.25, 0.5, 0.75, 1.0}; SDXL struggles with abstract QuickDraw references due to high-fidelity bias.
Inference time: ~40s (SD v1.5) and ~70s (SDXL) per sketch on A100 40GB.

Limitations¶

No localized style control—styles blend globally rather than being assignable to specific spatial regions.
SDXL backbone is incompatible with highly abstract reference datasets (QuickDraw), producing substantially lower style consistency (DINO-ref1 drops notably in QD-S5 pairs).
λ and ω₁/ω₂ require manual calibration per style domain (different defaults for dense vs. abstract styles).
Counter-based regulation (Sobel loss) trades stroke fidelity for artifact suppression, a non-zero cost.
Evaluation limited to CLIP-T, DINO, VGG Gram distance, and aesthetic score—no semantic sketch-specific metric.

Relevance to Vision-Language Models¶

M3S directly builds on VLM infrastructure: CLIP's cross-modal embedding space is used for text-alignment evaluation, and the diffusion backbone (SD/SDXL) leverages large-scale vision-language pretraining for zero-shot generalization to sketch domains. The work exemplifies a growing paradigm of repurposing VLM-backed generative models for non-photorealistic synthesis without additional training, exploiting the priors encoded during large-scale image-text pretraining. The style-content guidance formulation—splitting CFG into separate semantic and stylistic directions—is a technique with broader applicability to any text-conditioned VLM generation task that requires disentangled attribute control.