Aligning Generative Denoising with Discriminative Objectives Unleashes Diffusion for Visual Perception

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ADDP identifies and closes three gaps between generative diffusion denoising and discriminative perception objectives: uneven timestep contribution, training-denoising distribution shift, and lack of interactivity. Fixes are plug-and-play (no architectural changes) and improve depth estimation, referring image segmentation (RIS), and generalist perception models. A novel agentic workflow wraps the corrected diffusion model to enable multi-round, language-driven perception refinement.

Problem

Diffusion models repurposed for perception (depth estimation, RIS, generalist models) still treat all denoising timesteps uniformly, ignore distribution shift between training and inference trajectories, and discard the multi-step process's interactive potential. On multi-modal tasks like RIS, these gaps cause diffusion-based models to lag far behind discriminative baselines.

Method

Three complementary, architecture-agnostic interventions collectively called ADDP:

1. Contribution-aware timestep sampling. Define per-timestep contribution factors \(c_t^2\) derived either analytically from the DDPM coefficient \(c_t^2 \propto \frac{1-\bar\alpha_t}{\bar\alpha_t}\) or empirically via iterative \(R^2\) regression on held-out metric curves. Replace uniform \(t \sim \text{Uniform}\) with \(t \sim \text{Multinomial}(c_1^2, \ldots, c_T^2)\) so high-contribution steps receive proportionally more gradient signal.

2. Diffusion-tailored data augmentation. Corrupt ground truth \(x_0\) into \(x_0' = \text{Augment}(x_0, t)\) before forming noisy \(x_t\); augmentation intensity scales with \(t\) (larger deviation at larger \(t\)). For RIS: color/location/shape distortions; for depth: Gaussian blur. This simulates the exposure-bias drift that causes IoU to drop at late denoising steps.

3. Correctional guidance user interface. Re-purpose classifier-free guidance with a compositional score: $\(\tilde\varepsilon_\theta(x_t,t,D,D^-,I) = \varepsilon_\theta(\cdots,\phi_D,\phi_I) + w_I[\cdots] + w_D^-[\varepsilon_\theta(\cdots,D^-,I)-\cdots] + w_D[\varepsilon_\theta(\cdots,D,I)-\varepsilon_\theta(\cdots,D^-,I)]\)$ where \(D^-\) is a "correctional prompt" describing the error. An agentic workflow chains LLaVA (image captioning) → GPT-4o (top-\(k\) confusing object generation) → diffusion model (\(k\) corrected predictions) → majority vote.

Key Contributions

• Theoretical decomposition showing that the MSE between prediction and ground truth is proportional to \(\sum_t c_t^2 \|\varepsilon_t - \varepsilon_\theta(x_t,t)\|_2^2\), motivating weighted timestep sampling.
• Two practical estimators for \(c_t^2\): closed-form from diffusion schedule and data-driven \(R^2\) regression from validation metrics.
• Diffusion-tailored, task-specific data augmentation applied to \(x_0\)-prediction (shown superior to \(\varepsilon\)-prediction for this purpose).
• Correctional guidance formulation that composes multiple score functions to steer inference toward user-specified corrections.
• Proof-of-concept multi-round agentic workflow (LLaVA + GPT-4o + diffusion) validated at scale on RIS.

Results

• Depth (Marigold baseline): ADDP improves average rank from 2.4 → 1.4 across five zero-shot benchmarks; e.g., NYUv2 AbsRel 6.0→5.6, δ₁ 95.9→96.3; ETH3D AbsRel 7.1→6.3.
• RIS: ADDP enables a diffusion model to "catch up with some discriminative baselines for the first time" on RefCOCO/RefCOCO+/G-Ref (exact numbers truncated in provided text but described as closing the gap to discriminative encoder-decoder models like LAVT at ~72 IoU range).
• Generalist (InstructCV): Consistent improvements across depth, segmentation, and detection tasks.
• Agentic workflow with GPT-4o correctional prompts further boosts RIS IoU over single-round diffusion inference.

Limitations

• \(c_t^2\) estimation via \(R^2\) regression requires running inference on a validation set for each new perception task/model combination — not zero-cost to transfer.
• Correctional guidance agentic workflow is a two-round proof-of-concept; generalization to more rounds or failure recovery is unvalidated.
• Augmentation designs (blur for depth, color/shape/location for RIS) are task-specific and require domain knowledge to design.
• Approximation in the \(c_t^2\) derivation ignores trajectory drift errors (acknowledged as intractable).
• RIS experiments use InstructPix2Pix format; applicability to other diffusion architectures is implied but not exhaustively shown.

Relevance to Harnesses / Meta-Harnesses

ADDP's agentic workflow is a direct micro-harness: it chains a captioner (LLaVA), a language reasoner (GPT-4o), and a diffusion perception model in a fixed multi-step pipeline to iteratively refine predictions via generated correctional prompts. This is precisely the "orchestration of specialized models via a meta-controller" pattern that meta-harness researchers study. The broader insight — that the denoising process itself is a controllable interface amenable to external prompt injection — generalizes to any harness that wraps diffusion models as perception sub-agents and needs to steer them mid-inference rather than re-running from scratch.