Skip to content

\(\Delta \mathrm{Energy}\): Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalization

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces ΔEnergy, a novel OOD detection score for CLIP-based VLMs that measures the energy change when the top-c cosine similarities between image and text features are zeroed out. A companion fine-tuning objective (EBM) simultaneously maximizes a lower bound on ΔEnergy and induces domain-consistent Hessians, jointly improving both open-set OOD detection and closed-set covariate-shift generalization within a single prompt-tuning framework.

Problem

Existing VLM adaptation methods address OOD generalization (covariate shifts, known classes in new domains) and OOD detection (semantic shifts, unseen classes) independently. Joint approaches either require large additional compute (extra local prompts) or have been validated only on narrow post-hoc scoring functions. No prior work simultaneously and efficiently optimizes both objectives with formal guarantees.

Method

ΔEnergy (zero-shot OOD score): Given image features and K text features, the top-c cosine similarities are reset to zero (re-alignment). ΔEnergy = E₁ − E₀, where E₀ is the standard energy score before re-alignment and E₁ is recomputed after setting s̃ŷⱼ = 0 for j ∈ [1, c]. ID samples have consistently larger ΔEnergy than OOD samples (Theorem 3.2), and ΔEnergy provably achieves lower FPR than MCM under mild conditions (Theorem 3.3).

EBM (few-shot fine-tuning objective): During prompt tuning (only context vectors θ are learnable; both encoders frozen), a masking operation zeros image feature elements where (zI ⊙ zT)_j > 0, yielding a masked domain S'. The EBM loss L_ΔE = (1/N) Σ [E₂(xᵢ) − E₀(xᵢ)] is minimized alongside cross-entropy: L_EBM = L_CE + λ₀ · exp(L_ΔE). Theorem 3.4 shows minimizing L_ΔE maximizes the lower bound of ΔEnergy. Theorem 3.5 shows the local optimum satisfies |θᵀ(Ĥ_S − Ĥ_S')θ| ≤ ε·(1/N)Σ|θᵀ∇²_θ zT(xᵢ)θ|, i.e., domain-consistent Hessians, which Proposition 3.6 connects to an upper bound on the OOD generalization gap.

Key Contributions

  • ΔEnergy score: post-hoc, zero-shot OOD detector derived from energy change under vision-language re-alignment; theoretically shown to outperform MCM and vanilla energy scores.
  • EBM loss: a bound maximization objective that simultaneously tightens ΔEnergy lower bound and induces domain-consistent Hessians, bridging OOD detection and OOD generalization in one loss.
  • Unified prompt-tuning framework: lightweight (only context vectors updated), applicable in few-shot settings, no extra encoders or external data needed.
  • Theoretical guarantees: formal proofs for OOD discrimination (Th. 3.2–3.3), ΔEnergy lower-bound maximization (Th. 3.4), domain-consistent Hessians (Th. 3.5), and generalization gap bound (Prop. 3.6).

Results

  • Setup-I (ImageNet-1k, zero-shot): ΔEnergy achieves AUROC 87.10% vs. MCM 85.81%, ODIN 85.61%, CSP 78.84%; FPR95 46.40% vs. MCM 53.34% (best competitor).
  • Setup-I covariate vs. semantic OOD: ΔEnergy AUROC 78.68%, FPR95 67.16% vs. MCM 75.66% / 70.30%.
  • Setup-I (few-shot EBM, 16-shot, CLIP ViT-B/16): AUROC 81.90%, FPR95 65.90%, OOD ACC 63.28% — outperforms CRoFT (76.30% AUROC, 62.83% OOD ACC) by >5% AUROC.
  • Setup-II (cross-dataset, 16-shot): EBM achieves average FPR95 21.0–22.2 vs. next-best GalLoP 29.0; OOD ACC gains on VLCS (81.7% vs. GalLoP 81.3%, CRoFT 80.2%).
  • Overall headline: method outperforms recent approaches by 10%–25% in AUROC across benchmarks per abstract.

Limitations

  • Hyperparameter sensitivity: c (number of top similarities to zero), τ (temperature), p (masking proportion), and λ₀ all require tuning via validation sets.
  • Evaluated exclusively on CLIP ViT-B/16; generalization to other VLM architectures (e.g., larger ViT backbones, generative VLMs) is not demonstrated.
  • Theoretical guarantees for OOD generalization (Prop. 3.6) rely on a second-order Taylor expansion and assume bounded feature distance ε between source and target domains — tightness of this bound is not empirically characterized.
  • The masking strategy focuses on foreground-suppression, which may be domain-specific and may fail when foreground/background distinctions are ambiguous or inverted.
  • Hard OOD and conventional OOD results are relegated to the appendix; the main text does not report full ablation over design choices.

Relevance to Vision-Language Models

This work directly addresses a key deployment challenge for CLIP-style VLMs: distinguishing semantically novel inputs (open-set OOD) while remaining robust to domain shift (covariate OOD), a gap that standard prompt-tuning methods (CoOp, CoCoOp) fail to close jointly. The ΔEnergy score is a lightweight, training-free addition to any CLIP inference pipeline, complementing the growing literature on MCM-style VLM OOD detection (Ming et al., 2022; NegLabel; CSP). The EBM fine-tuning framework demonstrates that energy-based objectives can be coupled with Hessian-consistency theory to achieve simultaneous generalization gains — a principled connection that could extend to other multimodal alignment paradigms beyond CLIP prompt tuning.