Skip to content

Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformer

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ICEdit frames instruction-based image editing as in-context generation within a frozen Diffusion Transformer (FLUX.1 Fill), bypassing the need for architectural changes or large training corpora. A diptych prompt structure, a lightweight LoRA-MoE fine-tuning module, and a VLM-guided early-filter inference-time scaling strategy together achieve state-of-the-art editing quality using only 50K training samples—0.1% of prior methods' data requirements.

Problem

Fine-tuning-based editing methods require massive datasets (450K–10M pairs) and architectural modifications (channel concatenation, extra condition encoders), while training-free methods cannot follow complex instructions reliably. No prior work exploits large-scale DiT priors directly for instruction-guided editing without either extreme.

Method

In-Context Edit Paradigm: Images are arranged as a diptych (source left, target right); the edit instruction is embedded in a fixed-format prompt—"A diptych with two side-by-side images of the same scene. On the right, the scene is the same as on the left but {instruction}"—which bridges the gap between descriptive prompts and imperative editing commands. The inpainting DiT (FLUX.1 Fill) reconstructs the left half and generates the edited right half under a fixed binary mask; no new attention mechanisms or position encoders are added.

LoRA-MoE Fine-Tuning: A Mixture-of-Experts LoRA is inserted into the output projection layer of multimodal attention blocks. N parallel LoRA experts (rank 32, 4 experts, TopK=1) are routed by a classifier conditioned on visual tokens and text embeddings. Standard LoRA is applied elsewhere. Fine-tuning uses only 50K samples (9K MagicBrush + 40K OmniEdit).

Early Filter Inference-Time Scaling: M initial noise candidates are each denoised for m ≪ n steps. A VLM (Qwen2.5-VL-72B) performs pairwise bubble-sort comparisons on these early-step outputs to select the best seed, which is then fully denoised (n steps). This exploits the observation that instruction alignment is identifiable within the first few rectified-flow steps.

Key Contributions

  • In-context diptych editing paradigm for DiTs that requires no architectural modification and achieves ~70% GPT-score improvement over direct instruction prompting in training-free settings.
  • LoRA-MoE fine-tuning on 50K samples achieving 150% improvement in editing success over training-free baseline, with only 0.2B trainable parameters (1% of prior full fine-tune methods).
  • Early Filter Inference-Time Scaling using VLM pairwise comparison to select optimal noise seeds, yielding a 19% SC score increase and 16% overall VIE-Score boost.
  • SOTA performance on Emu Edit and MagicBrush benchmarks using 0.1% of training data of prior methods.

Results

  • Emu Edit test set (Table 1): ICEdit achieves CLIP-I 0.907, CLIP-Out 0.305, DINO 0.866, GPT 0.68—best GPT score among all open-source methods; beats UltraEdit (GPT 0.54), FluxEdit (GPT 0.22), EmuEdit (GPT 0.72 closed-source).
  • MagicBrush test set (Table 2): L1 0.060 (best), CLIP-I 0.928 (best), DINO 0.853; outperforms UltraEdit (L1 0.066, CLIP-I 0.904) and all DiT-based baselines.
  • VIE-Score: With inference-time scaling, ICEdit surpasses SeedEdit (commercial) in overall VIE-Score (78.2 vs. 75.7 without scaling).
  • Ablation (Table 3b): Early filter with 6 noise candidates and 10 early steps (NFE=74) achieves GPT 0.72 vs. 0.68 fixed-seed baseline; CLIP-based verifier (GPT 0.65) underperforms VLM verifier.
  • Data efficiency: 180% GPT-score increase over training-free framework at 0.05M samples vs. prior methods requiring 0.45M–10M.

Limitations

  • Inference-time scaling increases NFE significantly (e.g., 6 seeds Ă— 10 early steps + 50 full steps = 110 NFE vs. 50 NFE baseline), adding latency proportional to number of candidates.
  • VLM judge (Qwen2.5-VL-72B) adds a large external model dependency and compute overhead for noise selection.
  • Training-free diptych framework alone yields unsatisfactory layout preservation; fine-tuning is required for practical use.
  • MoE routing is assessed only with TopK=1 and 4 experts; scaling behavior with more experts is not studied.
  • Relies on FLUX.1 Fill as backbone; generalizability to other DiT architectures is not demonstrated.

Relevance to Vision-Language Models

VLMs play a direct and novel functional role here: Qwen2.5-VL-72B is used as an inference-time quality judge that evaluates whether early-denoising outputs satisfy edit instructions, replacing heuristic metrics like CLIP for noise selection. This is a clean example of VLMs acting as semantic verifiers in a generative pipeline rather than as generators themselves. The work also highlights the prompt-space gap between VLMs' descriptive language understanding and imperative editing instructions—a challenge shared across VLM-conditioned generation systems. For researchers tracking VLMs, this paper demonstrates a practical pattern: using a strong VLM to steer an image generation model without any joint training.