Skip to content

GMO-E\(^2\)DIT: Grounded Multi-Operation Editing for E-Commerce Images

🕒 Published (v1): 2026-07-01 13:23 UTC · Source: Arxiv · link

Why this paper was selected

Multi-operation grounded image editing for e-commerce; tests compositional VLM instruction following

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GMO-E²DIT is an agentic e-commerce image editing framework that couples a VLM planner/reflector with a mask-conditioned diffusion editor to execute compositional, localized multi-operation edits reliably. The VLM decomposes fuzzy instructions into a typed edit agenda, a deterministic compiler converts sub-programs into operation-aware masks, and a reflection loop assigns success/continue/rollback states after each round. It outperforms open-source and closed-source baselines on the new EComEditBench across instruction accuracy and edit fidelity.

Problem

One-shot instruction-driven editors conflate intent resolution, spatial grounding, operation binding, and rendering into a single generation step. For e-commerce images requiring multiple simultaneous local edits (e.g., move a sticker, exchange two patches, replace text color), this causes partial execution failures where some edits are correct and others are silently missed or erroneously applied — unacceptable in commercial scenarios. Users rarely supply spatial annotations, and existing VLM-guided methods treat reasoning only as auxiliary support around a fixed generator.

Method

GMO-E²DIT formulates editing as grounded multi-operation task completion. The system maintains a structured edit agenda \(\mathcal{U} = \{p_i\}_{i=1}^K\) where each entry \(p_i = (\tau_i, b_i^{\text{src}}, b_i^{\text{dst}}, a_i, h_i)\) specifies an operation type, source/target bounding boxes, attributes, and a textual hint.

At each round \(t\), a scheduler selects a sub-program \(Q_t\), and a deterministic compiler \(C\) converts it into a canonical prompt \(\tilde{u}_t\), operation-aware masks \(M_t\), and optional patch references \(A_t\). The mask-conditioned editor \(E_\theta\) generates the intermediate result: $\(\hat{y}_t = E_\theta(x_t, \tilde{u}_t, M_t, A_t)\)$

The same VLM then switches to reflection mode and assigns a state \(\hat{s}_t \in \{\text{success}, \text{continue}, \text{rollback}\}\). On success, execution stops; on continue, remaining tasks proceed from \(\hat{y}_t\); on rollback, the system reverts to the last safe image \(x_t\) and replans.

Training proceeds in three stages: (1) VLM SFT on planning and reflection data, followed by rule-based GRPO with verifiable rewards on format correctness, operation type/order matching, and bounding-box IoU; (2) mask-conditioned editor SFT using a flow-matching objective \(\mathcal{L}_{\text{edit}} = \mathbb{E}[\|v_\theta(z_t, t, x_t, \tilde{u}_t, M_t, A_t) - v^\star\|_2^2]\); (3) joint RL with the editor frozen, optimizing the VLM agent via GRPO with rewards from a stronger external VLM judge (Gemini-3 Pro), penalizing premature success decisions.

The data pipeline uses YOLOv10 + SAM2 for element detection/matting, PP-OCR-v5 for text extraction, and Qwen3.5 as a regional edit planner to synthesize aligned source–target pairs across six operation types, including multi-turn reflection triplets.

Key Contributions

  • Reformulation of e-commerce image editing as grounded multi-operation task completion with a role-aware typed edit agenda decoupling VLM reasoning from generative rendering.
  • GMO-E²DIT: a VLM agent (planner + reflector) coupled with a mask-conditioned editor in a reflection-driven closed loop with three-state (success/continue/rollback) error recovery.
  • Three-stage training regime combining SFT, rule-based GRPO, and joint RL with external VLM judge feedback at the program level rather than globally.
  • A unified e-commerce data synthesis pipeline (element mining → multi-task composition → reflection triplets) and EComEditBench, a 1,000-sample benchmark covering 5 atomic operations under both fuzzy and precise instruction protocols, scored by Instruction Accuracy (IA), Edit Fidelity (EF), and Background Preservation (BP).

Results

  • Fuzzy setting (best IA/EF overall): GMO-E²DIT achieves IA=8.66, EF=8.25, BP=8.87; vs. open-source Qwen-Image-Edit-2511 (4.52/4.12/4.91) and FLUX2-klein-base-9B (5.16/4.53/6.65).
  • Precise setting: IA=9.82, EF=9.62, BP=9.85; outperforms closed-source Nano Banana Pro (9.18/9.01/9.49) and Seedream 5.0 (9.29/9.04/9.49).
  • Role-sensitive operations (fuzzy): move scores 8.87/8.14/8.56 and exchange scores 9.07/8.54/9.30, demonstrating that operation-aware masks reduce source–target confusion.
  • Multi-operation composites (fuzzy): GMO-E²DIT leads IA and EF on both 2-operation and 4-operation cases.
  • Background preservation: second-best in the fuzzy setting; closed-source models (e.g., Nano Banana Pro) achieve higher BP despite lower instruction completion.

Limitations

  • Background preservation in the fuzzy setting is second-best; closed-source models better preserve non-target regions despite weaker instruction following.
  • Results table is incomplete in the provided text, precluding full per-operation comparison on all baselines.
  • The reflection module requires multiple editing rounds, increasing inference latency compared to one-shot editors; maximum rounds \(T\) is bounded but not ablated in the visible text.
  • The data pipeline and benchmark are confined to e-commerce poster/product imagery; generalization to other domains (natural photos, documents) is not evaluated.
  • Relies on a stronger external VLM judge (Gemini-3 Pro) for Stage 3 rewards and evaluation, introducing dependency on a closed-source model.

Relevance to Vision-Language Models

GMO-E²DIT directly demonstrates VLMs operating as both cognitive planners and reflective judges in a closed-loop agentic system — a concrete instantiation of VLM-based tool use and self-critique beyond passive captioning or VQA. The GRPO fine-tuning of the VLM agent on verifiable structured rewards (box IoU, operation-type correctness) is a practical example of RL post-training for grounded spatial reasoning, extending the emerging paradigm of reward-driven VLM alignment to multi-step visual tasks. The program-level reflection (per-operation success/continue/rollback) rather than global scoring is a methodological contribution relevant to anyone building VLM agents that must manage partial progress in multi-step workflows. EComEditBench also provides a new VLM-as-judge evaluation protocol for compositional visual editing that the broader VLM community can build on.