Skip to content

UniLumos: Fast and Unified Image and Video Relighting with Physics-Plausible Feedback

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

UniLumos is a unified image-and-video relighting framework built on a flow-matching backbone (Wan2.1) that injects RGB-space geometry feedback—depth and surface normal supervision—to enforce physical plausibility in lighting effects. It combines this with path consistency learning for few-step inference, achieving state-of-the-art relighting quality with a reported 20× (up to 76×) speedup over prior diffusion-based methods. A structured six-dimensional illumination annotation protocol and a VLM-driven benchmark (LumosBench) are introduced alongside the model.

Problem

Existing diffusion-based relighting methods operate in semantic latent space, where proximity does not guarantee physical correctness in RGB space, leading to artifacts such as overexposed highlights, misaligned shadows, and incorrect light–scene interactions. They also lack structured illumination descriptions and dedicated evaluation metrics; generic scores (FID, LPIPS) cannot capture shadow misalignment, intensity mismatch, or incorrect light direction. Additionally, standard multi-step denoising makes physics-guided RGB-space supervision computationally prohibitive.

Method

Architecture. UniLumos fine-tunes Wan2.1-T2V-1.3B (a flow-matching DiT) with concatenated conditioning: the degraded subject latent \(x_\text{deg}\), background latent \(x_\text{bg}\), and noisy latent \(x_t\) are fused along the channel dimension into DiT blocks. Text or reference-video conditions are encoded via umt5.

Physics-Plausible Feedback. After decoding predicted latents to RGB via Wan-VAE, a frozen dense estimator (Lotus) extracts depth \(\hat{D}\) and surface normals \(\hat{N}\). These are compared against pseudo-ground-truth maps from the input under a masked geometry loss: $\(\mathcal{L}_\text{phy} = \mathbb{E}\!\left[M \odot \left(\frac{\|\hat{D}-D\|_2}{\|D\|_2} + \frac{\|\hat{N}-N\|_2}{\|N\|_2}\right)\right]\)$

Path Consistency Learning. To make RGB-space supervision tractable under few steps, path consistency loss \(\mathcal{L}_\text{fast}\) enforces that two half-steps match one full step in velocity space, enabling 1–5 step inference without teacher-student distillation.

Joint Objective. \(\mathcal{L} = \mathcal{L}_0 + 0.1\,\mathcal{L}_\text{fast} + 0.1\,\mathcal{L}_\text{phy}\), with a batch scheduling strategy that allocates 20% to path-consistency (3 forward passes) and 50% of the remaining 80% to RGB geometry feedback.

LumosData. ~110K video pairs extracted from Panda70M plus 1.2M relit images via IC-Light, annotated with a six-dimensional protocol (direction, light source type, intensity, color temperature, temporal dynamics, optical phenomena) generated by Qwen2.5-VL.

LumosBench. VLM-based benchmark that predicts all six lighting attributes from generated outputs and compares them against targets for attribute-level controllability scoring, plus Dense L2 Error (depth/normal deviation) for geometric grounding.

Key Contributions

  • UniLumos model: unified image+video relighting via flow-matching with explicit RGB-space depth and normal feedback, achieving physical plausibility without geometry inputs at inference.
  • Path consistency learning integration: enables effective geometry supervision under 1–5 denoising steps, delivering 20–76Ă— inference speedup.
  • Structured 6D illumination annotation protocol: direction, light source type, intensity, color temperature, temporal dynamics, optical phenomena—used for both conditioning and evaluation.
  • LumosBench: disentangled attribute-level benchmark using VLMs (Qwen2.5-VL) for automatic, interpretable evaluation of relighting controllability.
  • LumosData pipeline: scalable data construction (110K video + 1.2M image pairs) from real-world sources with structured annotations.

Results

Image relighting (vs. SwitchLight, DiLightNet, IC-Light, SynthLight): - PSNR: 26.719 (best; SynthLight 25.572, IC-Light 24.316) - SSIM: 0.913 (best; SynthLight 0.905) - LPIPS: 0.089 (best; SynthLight 0.102) - Lumos Avg. Score: 0.912 (best; SynthLight 0.791) - Dense L2 Error: 0.103 (best; SynthLight 0.214)

Video relighting (vs. IC-Light per-frame, Light-A-Video+CogVideoX, Light-A-Video+Wan2.1): - PSNR: 25.031 (best; next best 20.784) - SSIM: 0.891 (best) - R-Motion: 1.436 (best temporal consistency; next best 1.582) - Lumos Avg. Score: 0.871 (best; next best 0.682) - Dense L2 Error: 0.147 (best; next best 0.371)

Inference speed: 12 s vs. 756–917 s for Light-A-Video variants on 49×832×480 video (~76× faster).

Ablation: removing all geometry feedback drops PSNR from 25.031 to 21.433; removing only normal feedback causes a larger drop than removing only depth, indicating normals are more critical for shadow alignment.

Limitations

  • Geometry feedback (Lotus) adds overhead during training (RGB decode + dense estimation per sample), requiring selective 80/20 batch scheduling to manage cost.
  • Pseudo-ground-truth depth/normal maps from the input image (under original illumination) may not perfectly match the target-relit geometry, introducing supervision noise.
  • Training only 5,000 iterations on 8Ă—H20 GPUs may limit coverage of complex multi-source or specular lighting scenarios.
  • LumosBench relies on VLM attribute prediction accuracy; errors in VLM lighting classification propagate into the benchmark score.
  • Evaluated primarily on human-centric scenes; generalization to complex object-centric or outdoor scenes is shown only in appendices (StanfordOrb, Navi).

Relevance to Vision-Language Models

VLMs play two direct roles in this work: Qwen2.5-VL is used at data construction time to auto-generate structured six-dimensional lighting captions for all training pairs, and the same VLM-based annotation pipeline is repurposed at evaluation time to score how well generated outputs match intended lighting attributes in LumosBench. This positions VLMs as both annotation engines and automatic evaluators for a visually grounded generation task—a pattern increasingly common in multimodal research where human labeling is too expensive. For researchers tracking VLMs, this paper illustrates a concrete workflow for using VLMs as structured perception modules to replace manual annotation and metric design in specialized visual domains.