Skip to content

GenAU: Language-Grounded Industrial Anomaly Understanding with Vision-Language Models

🕒 Published (v1): 2026-07-01 15:11 UTC · Source: Arxiv · link

Why this paper was selected

VLM for industrial anomaly detection+localization+explanation — practical grounded multimodal system

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GenAU is a unified instruction-following vision-language framework for industrial anomaly understanding that performs image-level detection, pixel-level segmentation, multi-type defect classification, and natural-language defect analysis within a single model. It extends an autoregressive VLM (LLaVA-OneVision) with two dedicated segmentation tokens whose decoder hidden states serve as language-grounded anchors aligned against multi-scale visual features. GenAU achieves the strongest image-level detection among CLIP-based zero-shot methods on VisA and Real-IAD while explicitly quantifying the segmentation–reasoning trade-off.

Problem

Existing zero-shot industrial anomaly detection methods based on CLIP excel at visual discrimination and localization but provide no semantic defect explanation; conversely, instruction-tuned VLMs can describe defects but do not natively produce pixel-level masks. No prior single-model approach jointly addresses all four practical inspection questions: anomaly existence, spatial localization, defect type, and interpretable reasoning.

Method

GenAU is built on LLaVA-OneVision (Qwen2 backbone + frozen SigLIP-so400m-patch14-384 encoder) at 0.5B and 7B scales. Two special tokens, [SEG_defect] and [SEG_normal], are added to the vocabulary; their final-layer decoder hidden states \(\mathbf{T} = [h_{\text{normal}}, h_{\text{defect}}] \in \mathbb{R}^{2 \times D}\) serve as language-grounded anchors. Multi-scale visual patch features \(\{V_i\}_{i=1}^{m}\) from the frozen encoder are projected per-stage via linear heads \(\phi_i\), then cosine-similarity logits are computed against \(\mathbf{T}\):

\[S_i = \gamma \cdot \frac{Z_i}{\|Z_i\|_2} \left(\frac{\mathbf{T}}{\|\mathbf{T}\|_2}\right)^\top, \quad Z_i = \phi_i(V_i).\]

A channel-wise softmax over upsampled \(S_i\) yields per-stage defect-probability maps \(A_i\), summed to a final anomaly map \(A\). An AnyRes high-resolution pathway reassembles tile-level patch features into a single grid for fine-grained localization, fused with the global view as \(A = (1-\beta)A_g + \beta A_{hi}\) (\(\beta=0.25\)). The image-level score is \(s_{\text{fused}} = \frac{1}{2}\tilde{s}_{\text{img}} + \frac{1}{2}\tilde{\ell}_{\text{nd}}\), blending the map maximum with the decoder's normal/defect log-odds calibrated once on MVTec. Training uses a joint objective \(\mathcal{L} = \mathcal{L}_{\text{txt}} + \lambda_{\text{seg}}\mathcal{L}_{\text{seg}}\), where \(\mathcal{L}_{\text{seg}}\) combines focal, Tversky (\(\alpha=0.3, \beta=0.7\)), and Dice losses across all encoder stages. Only LoRA (\(r=16\), \(\alpha=32\)) on the LLM, token embeddings/LM head, per-stage projectors, and decoder layer-norms are trained; encoder and cross-modal projector remain frozen.

Key Contributions

  • GenAU framework: single instruction-following VLM covering image-level detection, pixel-level segmentation, zero-shot multi-type detection, and textual defect analysis under one training recipe.
  • Language-grounded segmentation tokens: [SEG_defect]/[SEG_normal] hidden states act as anchors aligned directly with frozen multi-scale visual features—no external mask decoder required.
  • Fused image-level score: combines the segmentation map maximum with the decoder's normal/defect decision via frozen calibration estimated on MVTec; adds no new parameters.
  • AnyRes high-resolution readout: reassembles tile patch features into a full-resolution grid for fine-grained localization, fused with the global view.
  • Cross-dataset evaluation protocol: trained on MVTec-AD, evaluated zero-shot on VisA, MPDD, Real-IAD, and VisA-D&R, with explicit characterization of the segmentation–reasoning trade-off.

Results

  • VisA image-level detection: GenAU-7B achieves 87.6% AUROC and 89.9% AP, best among CLIP-based zero-shot methods (next best: AnomalyCLIP 82.1% AUROC); reasoning specialist Anomaly-OV-7B reaches 88.9% but uses a different protocol and produces no segmentation.
  • VisA pixel-level segmentation: GenAU-7B reaches 94.7% AUROC and 88.2% AUPRO; does not surpass strongest CLIP segmenter (FiLo: 95.9% AUROC).
  • MPDD: image-level AUROC 70.3% (below MultiADS-F 79.7%); pixel-level AUROC 94.5%, AUPRO 87.1%.
  • Real-IAD: image-level AUROC 84.7%, pixel-level AUROC 94.8%, AUPRO 87.1%.
  • Zero-shot multi-type detection: GenAU-7B achieves best F1 on both VisA (~29%) and MPDD (~70%), outperforming MultiADS and Anomaly-OV at both scales.
  • Ablations (VisA): removing LM fusion drops image-AUROC by 2.3 points; mixing reasoning data drops pixel-AUPRO by 2.6 points; removing LoRA costs 2.2 image-AUROC and 1.1 pixel-AUPRO.

Limitations

  • Pixel-level segmentation does not surpass specialized CLIP-based baselines (e.g., FiLo) on any evaluated dataset.
  • Image-level detection on MPDD (logic/large-area anomalies) lags MultiADS-F by ~9.4 AUROC points; global-context and missing-component defects remain challenging for the similarity-map readout.
  • Multi-type F1 is low on VisA (~29%), reflecting the difficulty of fine-grained defect taxonomies.
  • Trained on MVTec-AD test split (the only split with masks and type labels); the protocol is standard but limited to one source domain.
  • Calibration for the fused image-level score is estimated on the source domain (MVTec), which may not transfer optimally.
  • MultiADS multi-type comparison uses a fixed threshold of 0.5 on per-type channels, potentially understating that baseline.

Relevance to Vision-Language Models

GenAU demonstrates a concrete recipe for extending autoregressive VLMs beyond text-only grounding to pixel-level spatial localization by injecting task-specific segmentation tokens whose hidden states query frozen visual feature pyramids—a lightweight adaptation of the LISA reasoning-segmentation paradigm to a closed-domain inspection setting. For VLM researchers, it provides evidence that language-grounded token anchors can replace dedicated mask decoders for localization while preserving open-vocabulary reasoning, and explicitly quantifies the segmentation–reasoning trade-off incurred when adding language objectives to a localization model. The cross-dataset zero-shot protocol and the joint instruction-following + segmentation training recipe contribute to the broader effort of unifying perception and language in a single VLM pass, relevant to domains where both pixel-level evidence and natural-language explanation are required outputs.