Skip to content

One-for-All Few-Shot Anomaly Detection via Instance-Induced Prompt Learning

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

IIPAD is the first one-for-all few-shot anomaly detection method built on CLIP and BLIP-Diffusion. It replaces per-class fixed prompts with a class-shared generator that produces instance-specific prompts by fusing Q-Former object tokens with CLIP visual embeddings. A category-aware memory bank using Q-Former tokens for retrieval resolves the cross-class feature mismatch problem endemic to unified models.

Problem

Prior CLIP-based few-shot anomaly detection methods (WinCLIP, PromptAD) follow a "one-for-one" paradigm: they learn separate prompts per category, making them expensive to train and impractical at scale. One-for-all methods exist but require large normal-sample sets. No method simultaneously handles (1) few-shot data scarcity, (2) a single unified model across all categories, and (3) fine-grained anomaly localization.

Method

Instance-Specific Prompt Generator: A class-shared generator produces per-instance normal/abnormal prompts by (a) extracting Q-Former tokens Z_i from BLIP-Diffusion to get object tokens O^n_i, O^a_i via MLPs, and (b) computing instance-specific prefix tokens C^n_i, C^a_i from the CLIP class token via cross-attention over Q-Former outputs, then adding them to shared learnable prefix tokens [P_i]/[N_i] (CoCoOp-style). Final prompts are: S^n_i = [P_1+C^n_{i,1}]…[P_T+C^n_{i,T}][O^n_{i,1}]…[O^n_{i,m}].

Multi-Modal Prompt Training: Three losses train the generator: (1) semantic-level alignment (Ls) — contrastive pull of the normal prompt embedding toward V-V-attention-modified CLIP patch features, push of anomalous prompt away; (2) token-level alignment (Lf) — each prompt token is aligned to a weighted aggregate of its most-similar patches via sparse attention; (3) textual guidance (Lt) — contrastive alignment of learned prompts to manually crafted general normality/abnormality descriptions (using "object" as a stand-in category name). Synthetic anomalous features for the abnormal prompt are constructed by fusing features from adjacent CLIP layers plus Gaussian noise.

Category-Aware Memory Bank: Stores patch tokens F_ij alongside Q-Former tokens Z_i. At test time, retrieval is two-stage: first match by Q-Former token similarity (category gate), then by patch feature similarity — preventing normal features from one category from masking anomalies in another. A parallel prompt embedding bank stores training prompt embeddings, retrieved similarly and averaged with the test-time generated prompt.

Inference: Final anomaly score = prompt-guided map Āp + visual memory map Av; image-level score uses the sum of per-map maxima.

Key Contributions

  • First one-for-all few-shot anomaly detection method leveraging VLMs (CLIP + BLIP-Diffusion).
  • Instance-specific prompt generator that generates exclusive per-image normal/abnormal prompts without any per-class learned parameters.
  • Token-level prompt–patch alignment loss that improves fine-grained localization.
  • Synthetic anomaly feature construction from multi-layer CLIP feature fusion for anomalous prompt supervision.
  • Category-aware two-stage memory bank retrieval that eliminates cross-class mismatch in unified models.

Results

  • MVTec 1-shot (one-for-all): IIPAD 94.2 AUROC-I / 96.4 AUROC-P vs. InCTRL 94.2/96.4, PromptAD 86.3/91.8, WinCLIP 92.8/92.4.
  • VisA 1-shot (one-for-all): IIPAD 85.4 AUROC-I / 96.9 AUROC-P vs. InCTRL 85.4/96.9, PromptAD 80.8/96.3.
  • MVTec 4-shot (one-for-all): IIPAD 96.1 AUROC-I / 97.0 AUROC-P; outperforms full-shot UniAD (96.5/96.8) at pixel level and surpasses full-shot OmniAL/DiAD on VisA.
  • One-for-one comparison (Table 2): IIPAD under one-for-all paradigm achieves pixel-level AUROC-P ≥ all one-for-one baselines at 2-shot/4-shot on MVTec (96.7, 97.0) and matches/exceeds PromptAD on VisA.
  • Ablation: Removing category-aware memory drops MVTec AUROC-P from 96.4 to 95.9; removing textual guidance drops AUROC-I from 94.2 to 91.4 (MVTec 1-shot).

Limitations

  • Requires BLIP-Diffusion Q-Former at inference, increasing model complexity and memory footprint beyond standard CLIP-only pipelines.
  • Synthetic anomaly features (layer-fused CLIP activations + Gaussian noise) are a heuristic; no guarantee they cover real anomaly distributions.
  • Memory bank scales linearly with training set size; behavior at large scale is not evaluated.
  • InCTRL, which also achieves competitive image-level AUROC, is noted to "struggle with anomaly localization" — but IIPAD's margin over InCTRL at image level is narrow (≤1 AUROC point in several settings).
  • Evaluated only on MVTec and VisA; generalization to medical imaging or non-industrial domains is untested.

Relevance to Vision-Language Models

IIPAD demonstrates a principled use of CLIP's joint embedding space for fine-grained discriminative tasks beyond classification — specifically industrial anomaly detection — by reformulating static text prompt learning into a dynamic, instance-conditioned generation problem. It is a concrete example of how Q-Former-style vision-language bridging modules (from BLIP-Diffusion) can inject fine-grained visual grounding into CLIP's text branch, complementing the V-V attention mechanism for local feature preservation. The paper's multi-modal training objective (visual contrastive + token-level patch alignment + textual expert knowledge) offers a reusable recipe for adapting VLMs to one-for-all settings where class identity is unavailable at test time. For VLM researchers, the category-aware retrieval strategy is particularly relevant as a scalable solution to embedding-space collision in multi-domain CLIP applications.