Skip to content

EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editing

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; EditReward: human-aligned reward model for instruction-guided image editing

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EditReward is a human-aligned reward model for instruction-guided image editing, trained on a new 200K expert-annotated preference dataset (EditReward-Data). It outperforms GPT-5 and other VLM-as-judge baselines on established editing benchmarks, and demonstrates utility as a data curator to improve downstream editing models.

Problem

Open-source instruction-guided image editing models lag behind closed-source systems (GPT-Image-1, Seedream) primarily because existing reward signals for synthetic data filtering are poorly aligned with human preferences. Perceptual scores (LPIPS) miss semantic accuracy, CLIP-based features miss editing semantics, and general-purpose VLM-as-judge models are not optimized for this task. Prior fine-tuned reward models rely on noisy crowdsourced or model-generated pseudo-labels, producing insufficient signal to scale high-quality training data.

Method

Dataset construction (EditReward-Data): 9,557 instruction–image pairs from six editing benchmarks are processed through six state-of-the-art editing models (Step1X-Edit, Flux-Kontext, Qwen-Image-Edit, BAGEL, Ovis-U1, OmniGen2) with multiple random seeds, yielding 12 candidates per pair. Expert annotators score seven randomly sampled candidates per pair on a 4-point Likert scale along two dimensions: Instruction Following (IF) and Visual Quality (VQ), producing >200K preference pairs with Krippendorff's \(\alpha = 0.668\) (IF) and \(0.597\) (VQ).

Model architecture: A VLM backbone (\(\mathcal{H}_\psi\), either Qwen2.5-VL-7B or MiMo-VL-7B) encodes the source image \(I_s\), editing prompt \(P\), and edited image \(I_{e,i}\); an MLP reward head \(\mathcal{R}_\omega\) maps the representation to a scalar: \(s_i = \mathcal{R}_\omega(\mathcal{H}_\psi(I_s, P, I_{e,i}))\).

Multi-Dimensional Uncertainty-Aware Ranking Loss: Extending HPSv3, the reward head outputs per-dimension Gaussian distributions \(s_{i,d} \sim \mathcal{N}(\mu_{i,d}, \sigma_{i,d}^2)\) for \(d \in \{1,2\}\) (IF, VQ). Dimensional means are aggregated via pessimistic minimum, balanced average, or direct summation into \(\mu_i^{\text{agg}}\), and a probabilistic preference probability \(P(I_h \succ I_l)\) is trained with negative log-likelihood: $\(\mathcal{L}_{\text{rank}} = -\log(P(I_h \succ I_l))\)$ An alternative regression loss minimizes MSE between the aggregated predicted mean and transformed ground-truth sum scores.

Tie disentanglement: Tied pairs where one image is better on IF and the other on VQ are decomposed into two opposing-label training samples, doubling the utility of tie data and improving training stability.

Data curation application: EditReward scores 46K examples from ShareGPT-4o-Image (at ~0.25 s/sample) and selects a top-20K subset to fine-tune Step1X-Edit, evaluated on GEdit-Bench.

Key Contributions

  • EditReward-Data: 200K expert-annotated pairwise preference pairs covering 12 editing categories from 7 models, with multi-dimensional (IF + VQ) 4-point Likert scoring and verified inter-annotator agreement.
  • EditReward: VLM-based reward model with Multi-Dimensional Uncertainty-Aware Ranking Loss that achieves state-of-the-art human alignment on GenAI-Bench, AURORA-Bench, and ImagenHub.
  • EditReward-Bench: New evaluation benchmark with 2/3/4-way multi-candidate preference ranking tasks (K=2/3/4), providing harder discrimination than existing pairwise benchmarks.
  • Tie-disentanglement strategy: Novel augmentation that decomposes overall-tied pairs into dimension-specific preference labels, extracting richer training signal.
  • Demonstrated data curation utility: EditReward-filtered 20K subset lifts Step1X-Edit's GEdit-Bench overall score from 6.7 to 7.1 (on par with Doubao-Edit), showing reward-driven filtering outperforms training on the full 46K set.

Results

  • GenAI-Bench (accuracy): EditReward (MiMo-VL-7B) 65.72 vs. GPT-5 59.61, ADIEE 59.96, Gemini-2.5-Flash 57.01, base Qwen2.5-VL-7B 40.48 (+23.24 pts from fine-tuning).
  • AURORA-Bench (accuracy): EditReward (MiMo-VL-7B) 63.62 vs. OpenAI GPT-4o 50.81, GPT-5 47.27, ADIEE 55.56.
  • ImagenHub (Spearman correlation): EditReward (Qwen2.5-VL-7B) 36.18 (second best behind GPT-4o 38.21); MiMo variant 35.20.
  • EditReward-Bench overall (K=2,3,4 combined): EditReward (MiMo) 38.42 vs. GPT-5 37.81, Gemini-2.5-Flash 38.02.
  • GEdit-Bench (data curation): Step1X-Edit + Top-20K subset achieves GO=7.895 (EN-Intersection) vs. Step1X-Edit + full ShareGPT-4o-Image 6.803 and base Step1X-Edit 7.131; roughly on par with Doubao-Edit (7.137).
  • Scoring throughput: 0.25 s/sample, ~2.61 GPU-hours for 46K samples.

Limitations

  • EditReward-Bench and EditReward-Data originate from the same annotation pool; held-out splits mitigate but do not fully eliminate potential distribution overlap.
  • Only two evaluation dimensions (IF, VQ) are scored; other quality axes (compositional coherence, identity preservation) are not explicitly modeled.
  • The optimal subset size (20K) is task- and dataset-specific; transferability of this hyperparameter to other noisy pools is untested.
  • Training images are resized to \(448 \times 448\), which may disadvantage evaluation of high-resolution or detail-sensitive edits.
  • The backbone requires 7B-parameter VLMs plus MLP heads; inference cost may be nontrivial for very large-scale data curation compared to CLIP-based filters.
  • Failure mode analysis (Appendix A.13) is acknowledged but not described in the main text.

Relevance to Vision-Language Models

EditReward directly advances VLM-as-judge methodology by demonstrating that task-specific fine-tuning with high-quality human preference data dramatically outperforms general-purpose VLMs (including GPT-5) for image editing evaluation — a +23-point gap from Qwen2.5-VL-7B to its fine-tuned variant on GenAI-Bench. The multi-dimensional uncertainty-aware ranking loss is a principled extension of reward modeling paradigms (Bradley-Terry, HPSv3) to multi-attribute, tri-modal inputs (\(I_s\), \(P\), \(I_e\)), directly relevant to reward models for RLHF/RLAIF training of generative VLMs. It also provides a well-annotated benchmark (EditReward-Bench) for measuring progress in VLM-based evaluation of instruction following under multi-way preference conditions, where current frontier models still struggle. The work establishes a scalable data flywheel: reward model → curated data → stronger editing model → better reward training candidates.