FlowMimic: Mask-free Visual Editing and Generation with Pixel-pair Warped Flow Field for Online Video Editing Data Generation and Modality Mimicry

🕒 Published (v1): 2026-07-20 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FlowMimic introduces a mask-free, scalable paradigm for video editing that generates video training pairs in real time from existing image editing samples using a pixel-pair 4D temporal warped flow field, eliminating the need for curated, mask-annotated video data. The system post-trains on Wan2.1-T2V-1.3B and adds modality mimic losses to align image and video output distributions. A set of sense-related tasks and region-aware losses are used to internalize editing-region localization without external masks or MLLM modules at inference.

Problem

Collecting video editing training data requires labor-intensive pipelines: object mask annotation via VLMs/SAM, I2V-model-based pair synthesis (which introduces artifacts), and VLM- or RL-based quality filtering. These pipelines are task-specific, poorly scalable, and produce far narrower task diversity than image editing datasets. Additionally, joint image-video models suffer from misaligned output distributions between the two modalities (T2I outputs look less photorealistic than T2V first frames), and language-guided editing-region localization relies on external masks or fine-tuned MLLM modules at inference time.

Method

Pixel-pair 4D Temporal Warped Flow Field. Given an image editing pair (source, target), FlowMimic constructs a 3D canonical grid for each and builds a pixel-pair temporal warped flow field via grid inverse sampling. Each spatially corresponding pixel pair is allowed to "travel" synchronously through time, preserving pixel-level editing correspondence across all frames while permitting free (even unnatural) spatial motion. Video editing samples are generated online from image editing samples alone — no curated video pairs are used.

Modality Mimic Losses. An image is treated as a single-frame special case of video. Two auxiliary losses are defined on one-step reversed denoised predictions: - Modality mimic generation loss \(\mathcal{L}_{\text{mimic,gen}}\): encourages T2I generation to mimic the first frame of T2V, improving photorealism. - Modality mimic editing loss: encourages V2V editing to mimic the (easier, faster-converging) I2I editing distribution.

Editing-Region-Aware Learning. Rather than providing an explicit mask at inference or integrating a frozen/fine-tuned MLLM, FlowMimic introduces sense-related tasks (referring expression segmentation, sense select) with corresponding editing-region-aware losses at the latent level and attention level, internalizing region localization into the model weights. A reference-inject self-attention mask is added to the base architecture. The backbone is Wan2.1-T2V-1.3B (DiT-based, flow matching, 3D spatio-temporal VAE encoding \(S_V \in \mathbb{R}^{(4N+1)\times H \times W \times C}\) to latent \(z \in \mathbb{R}^{(N+1)\times\lfloor H/8\rfloor\times\lfloor W/8\rfloor\times 16}\)).

Key Contributions

• Pixel-pair 4D temporal warped flow field: real-time online generation of video editing pairs from image editing samples alone, requiring no mask annotation or I2V synthesis pipeline.
• Demonstration that temporal consistency of pixel-level editing correspondences suffices for a model to learn video editing (human-intelligible temporal coherence is not required in training).
• Modality mimic generation loss and modality mimic editing loss to align T2I/T2V and I2I/V2V output distributions.
• Internalized editing-region-aware localization via sense-related tasks and latent/attention-level losses, removing inference-time mask or MLLM dependencies.
• Extension of the pixel-aligned data paradigm to high-level video tasks (referring expression segmentation, semantic segmentation) and low-level tasks (video deblurring).

Results

• FlowMimic demonstrates effective mixed-task video editing (relighting, stylization, head swapping, object insertion, virtual try-on, referring expression segmentation) after joint training exclusively on online-generated video samples.
• Compared to the pretrained Wan2.1 T2IV baseline, FlowMimic produces more camera-authentic T2I results, and the image/video output distributions are better aligned.
• The model begins responding to video editing tasks within a few thousand training steps at a learning rate warmed from \(5\times10^{-6}\) to \(1\times10^{-5}\).
• Cross-attention map analysis (Fig. 28) shows reliable localization of textual phrases to corresponding visual regions in reference input.
• The model achieves SAM-like temporal object tracking via referring expression segmentation (e.g., tracking "a girl in yellow" across video frames) using only online-generated pixel-aligned data.
• (Note: the paper does not report quantitative benchmark numbers in the provided text; results are qualitative and ablative.)

Limitations

• No quantitative comparison against state-of-the-art methods on standard video editing benchmarks is reported in the available text.
• The pixel-pair warped flow field produces temporally unnatural deformations; the model must generalize from these synthetic motions to real-world coherent videos at inference — this generalization is asserted but not quantitatively verified across motion regimes.
• Experiments use a lightweight 1.3B-parameter backbone; scalability to larger models is not evaluated.
• The approach requires "loose layout correspondence" between source and target image editing pairs for a given task — tasks lacking this property may not transfer.
• The paper acknowledges lossy compression of figures in the arXiv version, potentially limiting figure-based evidence.

Relevance to Vision-Language Models

FlowMimic directly engages VLMs as components in its framing: existing video editing pipelines use VLMs (SAM, DINO, Qwen-VL) for mask annotation and quality filtering, which FlowMimic aims to replace with a scalable data generation paradigm. More significantly, FlowMimic internalizes the cross-modal text-visual localization capability that competing methods delegate to fine-tuned MLLMs or external mask sequences, demonstrating that sense-related tasks with region-aware losses can embed VLM-like grounding directly into a generative model. For researchers tracking VLMs, this represents a relevant data efficiency angle: the pixel-aligned paradigm extends to referring expression segmentation, showing that VLM-style language-grounded tasks can be learned from synthetically warped video data without real grounded video corpora.