Do Image Editing Models Understand Lighting?

🕒 Published (v1): 2026-06-25 08:20 UTC · Source: Arxiv · link

Why this paper was selected

Nießner+Rother at ECCV; probes fundamental lighting understanding gap in generative VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces 3DLP, a benchmark and 1K HDR real-world image-pair dataset for evaluating whether instruction-based image editing models correctly simulate physical light transport when toggling a visible light probe. Two new metrics—Standardised Intensity Error (SIE) and Low-Frequency Error (LFE)—are designed to be invariant to AI-generated global photographic adjustments. Evaluation of six state-of-the-art models shows large performance gaps, with commercial models leading and all models struggling in low-illumination regions.

Problem

Existing benchmarks for image editing assess lighting plausibility via VLMs or human judgement, which are optimised for perceptual realism rather than radiometric accuracy, and thus cannot quantify pixel-level physical correctness of light transport effects. Synthetic datasets and OLAT captures cover isolated objects or lack scene diversity. There is no high-fidelity, real-world benchmark that objectively measures whether generative editing models truly understand illumination physics (inverse-square falloff, Lambertian shading, specular highlights, cast shadows, inter-reflections).

Method

The benchmark anchors a simple spherical light probe (floor lamp with visible bulb and base) in real indoor scenes. For each scene, HDR image pairs are captured with the probe on and off using 9-exposure brackets (\(\text{EV step} = 1.0\)) averaged to reduce noise and merged to linear HDR.

To isolate the light probe's contribution, the method computes ratio images: $\(E^{\text{on}}_{\text{AI}} = I^{\text{on}}_{\text{AI}} / I^{\text{off}}_{\text{R}}, \quad E^{\text{on}}_{\text{R}} = I^{\text{on}}_{\text{R}} / I^{\text{off}}_{\text{R}}\)$

Under the formulation \(I^{\text{on}}_{\text{AI}} = C \odot L_R + K \odot I^{\text{off}}_{\text{R}}\) (where \(C\) is unknown AI light color/intensity and \(K\) is global white-balance/exposure shift), the ratio images satisfy \(E^{\text{on}}_{\text{AI}} = C \odot E^{\text{on}}_{\text{R}} + K - C\)—a global per-channel affine relationship—so metrics are made invariant to both \(C\) and \(K\) via robust standardisation using median and MAD:

\[S(X) = \frac{X - \text{median}(X)}{\text{MAD}(X)}\]

SIE (Standardised Intensity Error): \(\text{SIE}^t = \text{MAE}(S(E^t_R), S(E^t_{\text{AI}}))\) — measures intensity prediction accuracy.

LFE (Low-Frequency Error): applied to Sobel gradient magnitudes masked at the 80th-percentile edge threshold, measuring spatial light-falloff and Lambertian shading accuracy while ignoring texture edges.

Annotation covers cast shadows, specular highlights, ambient highlights/shadows, and challenging materials (metallic, mirror, transparent), covering 10.92% of pixels. Evaluation uses the best 80% of images per model to exclude task-misinterpretation failures.

Key Contributions

• 3DLP dataset: 1K HDR image pairs of diverse real-world indoor scenes with physically toggled light probes; 14-bit 9-exposure brackets, linear HDR output with EXIF metadata.
• Fine-grained annotations: polygon-labeled regions for cast shadows, specular highlights, ambient effects, and challenging materials (metallic, mirror, transparent).
• SIE and LFE metrics: physically motivated, invariant to AI-applied global exposure and white-balance shifts, enabling objective pixel-level comparison against real ground truth.
• Systematic evaluation of six state-of-the-art commercial and open-source image editing models (Nano Banana Pro, Nano Banana 2, Qwen-Image-Edit, Flux 2 Max, Flux 2 Dev, GPT Image 1.5).
• Finding that VLMs are unsuitable for pixel-level light transport evaluation despite their utility for macroscopic plausibility assessment.

Results

• Nano Banana Pro ranks 1st overall (Avg. Rank 1.0); Nano Banana 2 is 2nd (1.8); Qwen-Image-Edit best open-source (3.3).
• Turn-on SIE: Nano Banana Pro 0.813, Nano Banana 2 0.882, Qwen-Image-Edit 1.239, Flux 2 Max 1.571, Flux 2 Dev 1.747, GPT Image 1.5 2.336.
• Turn-off SIE: Nano Banana Pro 0.769, Nano Banana 2 0.871, Qwen-Image-Edit 0.982, GPT Image 1.5 1.887, Flux 2 Max 1.303, Flux 2 Dev 2.136.
• LFE scores are closer across models than SIE scores, especially for specular highlights.
• All models show monotonically increasing SIE error as light intensity band decreases (farther from source), confirming failure at low-illumination regions.
• Mirrors show the largest inter-model spread; specular highlights show the smallest.
• VLMs tested on the task are found unsuitable for pixel-level light transport quantification.

Limitations

• Only six models evaluated; task-specific relighting models (which may perform better) were excluded due to unavailability of code/weights.
• Benchmark restricted to a single simple lamp geometry (spherical bulb); complex or non-spherical luminaires are known to reduce metric reliability (shown experimentally).
• Only 10.92% of pixels carry region annotations; coverage is not exhaustive.
• Evaluation uses 80% of images per model, discarding failure cases, which may underestimate real-world error rates.
• The benchmark is indoor only and cannot characterize outdoor or time-varying natural illumination.
• \(C\)-invariance for the turn-off task allows a model that only dims (rather than fully extinguishes) the lamp to avoid penalty, though this was not observed empirically.

Relevance to Vision-Language Models

This paper directly benchmarks VLMs as evaluators for light transport and finds them inadequate for pixel-level radiometric assessment—a significant negative result for the VLM-as-judge paradigm widely used in image editing benchmarks. It exposes a concrete capability boundary: VLMs capture perceptual/semantic plausibility but fail at quantitative physical consistency, which has direct implications for how the community should design evaluation pipelines for VLM-driven editing systems. For researchers tracking VLMs, the 3DLP benchmark provides a challenging physically grounded probe that can differentiate multimodal models (e.g., Qwen-Image-Edit, GPT Image 1.5, Flux 2) on a property—physical light transport—not covered by semantic faithfulness or instruction-adherence benchmarks.