Skip to content

Language-Assisted Feature Transformation for Anomaly Detection

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LAFT is a training-free feature transformation method that uses CLIP's shared image-text embedding space to suppress or amplify specific visual attributes in anomaly detection. By projecting image features onto (or orthogonal to) language-derived "concept subspaces," users can direct what the detector treats as anomalous without additional data or fine-tuning. It consistently outperforms prior CLIP-based anomaly detection baselines on both semantic and industrial benchmarks.

Problem

Unsupervised anomaly detection methods model normality purely from data, making them unable to incorporate user intent: they flag any deviation from training distribution, including irrelevant nuisance attributes (e.g., background in Waterbirds, color in MNIST), and fail when training data is biased or incomplete. Existing user-guided methods either require anomaly labels/samples or demand normality to be fully expressible in text prompts alone.

Method

LAFT operates entirely within CLIP's shared embedding space without additional training:

  1. Concept subspace construction: Users supply text prompts enumerating concept values for a target attribute (e.g., "circle," "square" for shape). CLIP text encoder produces embeddings; all pairwise differences between prompt embeddings are computed, then PCA extracts the top-d principal axes — the "concept axes" — to remove template noise.

  2. Feature transformation: For guide mode (emphasize relevant attribute), image features are projected onto the concept subspace (Eq. 5). For ignore mode (suppress nuisance attribute), image features are projected orthogonal to the concept subspace (Eq. 6).

  3. Anomaly scoring: Transformed features feed a k-NN classifier (k=30, cosine similarity) for semantic anomaly detection (LAFT AD). For industrial anomaly detection, LAFT is integrated into WinCLIP's multi-scale window/image/text embeddings (WinCLIP+LAFT).

The method requires only a user-supplied list of concept value strings; no labels, no anomaly samples, no gradient steps.

Key Contributions

  • LAFT: training-free image feature transformation via CLIP text-derived concept subspaces (projection/orthogonal projection)
  • LAFT AD: semantic anomaly detector combining LAFT with kNN scoring
  • WinCLIP+LAFT: integration of LAFT into WinCLIP for industrial anomaly detection
  • Demonstrated robustness to prompt quality (partial or even unseen concept values still perform near-optimally)

Results

  • Colored MNIST (Number attribute): LAFT AD (Guide) achieves 98.5% AUROC / 98.4% AUPRC / 6.9% FPR95, vs. best prior method InCTRL at 94.0% AUROC and WinCLIP+ at 92.6% AUROC
  • Waterbirds (Bird attribute): LAFT AD 95.6% AUROC vs. WinCLIP+ 91.8% and ZOE 92.2%
  • CelebA (Hair color): LAFT AD 95.0% AUROC vs. WinCLIP+ 92.8%; CelebA (Eyeglasses): LAFT AD 98.1% AUROC vs. InCTRL 87.8%
  • MVTec AD (K=8 few-shot): WinCLIP+LAFT-C 96.5% AUROC vs. WinCLIP+ 95.7% and InCTRL 93.8%
  • VisA (K=8 few-shot): WinCLIP+LAFT-C 88.3% AUROC vs. InCTRL 89.4%, WinCLIP+ 88.2%
  • Ablation: performance degrades gracefully with reduced prompt quality; auxiliary unseen concept values cause minimal harm

Limitations

  • Requires the user to manually specify which attributes are relevant or irrelevant — not applicable when the user has no prior knowledge
  • Normality is still primarily defined by training images; LAFT only refines boundaries, so it cannot compensate for severely biased training sets
  • Ignore mode (orthogonal projection) is fundamentally harder than guide mode and uses far more PCA components (32–384 vs. 8–32), with generally weaker invariance
  • For industrial anomaly detection, LAFT cannot replace multi-scale spatial features; it only augments WinCLIP rather than standing alone
  • Prompt sensitivity: performance depends on supplying actual concept values rather than attribute names; incorrect or entirely missing values degrade results

Relevance to Vision-Language Models

LAFT is a direct application of CLIP's cross-modal alignment geometry: it treats the shared embedding space as a semantic manifold where text-derived directions correspond to interpretable visual attribute axes — an approach that only works because VLMs learn aligned multimodal representations. It deepens understanding of how CLIP embeddings encode compositional visual concepts (echoing word2vec arithmetic), and shows that VLM representations are geometrically structured enough to support zero-shot feature surgery. For the VLM tracking community, it is a concrete example of using frozen VLM embeddings as a controllable interface for downstream perception tasks, complementing prompt-tuning and adapter-based lines of work by demonstrating that projection alone can steer model behavior.