Skip to content

Now You See the Hate: Adaptive View Retrieval for Hidden Hateful Illusions

🕒 Published (v1): 2026-07-21 12:50 UTC · Source: Arxiv · link

Why this paper was selected

VLM safety gap: state-of-the-art models under 25% accuracy on hateful optical illusions

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Current VLMs and moderation classifiers fail catastrophically on hateful optical illusions, with nine state-of-the-art VLMs (including GPT-4o and Gemini-2) achieving ≤10.2% accuracy on original-view hidden hate. This paper reframes the problem as perceptual retrieval: construct a multi-view bank of classical image transforms, adaptively weight views per image, and retrieve hidden-message templates before issuing a harmful/benign decision. With a frozen CLIP encoder, Adaptive View Retrieval reaches 93.2% balanced accuracy on HatefulIllusion.

Problem

Hateful optical illusions embed slurs and extremist symbols into benign-looking scenes using Gestalt perceptual mechanisms (figure-ground separation, contour closure). Standard moderation pipelines classify the surface view and thus have no perceptual access to the hidden layer. No single classical image transform is universally effective—hate slangs, hate symbols, and varying visibility levels each demand different perceptual filters, making fixed-filter rules unreliable.

Method

Adaptive View Retrieval is a retrieve-and-calibrate architecture built on a frozen CLIP ViT-B/32 encoder.

View bank construction: Seven deterministic, classical transforms are applied to both the query image and every hidden-message template: (1) original view, (2) contrast enhancement via luminance histogram equalization in YCbCr space, (3) closure-edge view (grayscale → edge detection → morphological closing), (4–6) three figure-ground views (mask, inverse-mask, foreground composite via Otsu thresholding), and (7) a low-pass view (cascaded Gaussian + box + median blur → grayscale → contrast normalization).

Adaptive gating and retrieval: For each branch \(k\), the frozen CLIP encoder extracts features \(e_x^{(k)}\) and \(e_j^{(k)}\), which branch-specific MLPs project to unit-norm embeddings \(z_x^{(k)}\) and \(z_j^{(k)}\). A gate network reads the concatenated pre-projection query features and outputs softmax weights \(\alpha_k(x)\). Template \(m_j\) is scored by:

\[S_j(x) = \tau \sum_{k=1}^{K} \alpha_k(x)\, \langle z_x^{(k)},\, z_j^{(k)} \rangle, \quad \hat{m} = \arg\max_{m_j \in \mathcal{M}} S_j(x)\]

Calibration: A separate MLP reads the gated fusion \(\bar{z}(x) = \sum_k \alpha_k(x) z_x^{(k)}\) and outputs a harmful/benign logit \(d(x)\), decoupled from the retrieved template.

Training: Joint cross-entropy retrieval loss (weight 1.0), binary cross-entropy calibration loss (0.5), and a gate-entropy regularizer (0.01) to prevent view collapse. Only branch projections, gate, calibration head, and temperature \(\tau\) are trained; CLIP remains frozen.

Key Contributions

  • Adaptive View Retrieval: a retrieve-and-calibrate framework that decouples hidden-message identity recovery from harmful/benign moderation.
  • Demonstration that adaptive view weighting—not any single classical transform—is necessary for reliable hidden-hate detection across target types and visibility levels.
  • Generalization to open-domain illusion recognition: surpasses fine-tuned CLIP baselines and human performance on IllusionMNIST, IllusionFashionMNIST, and IllusionAnimals.
  • Under the SemVink VLM QA protocol on HC-Bench, multi-view questioning with the same view bank achieves 98.2% overall accuracy vs. 17.0% for the original image and 2.7% for zoom-out.

Results

  • HatefulIllusion (main benchmark): 93.2% ± 3.1% balanced accuracy (frozen CLIP), vs. ≤20.9–24.5% for six moderation classifiers and ≤10.2% for nine VLMs with illusion-aware prompting on original-view inputs.
  • Trained original-view retrieval ablation: 79.5% on full test set, 78.9% on hard subset; adaptive multi-view retrieval adds +13.7 pp and maintains the gap on hard cases.
  • Filter-only baselines: ≤~60% balanced accuracy on full test set; remain weak on hard subset.
  • IllusoryVQA Top-1 accuracy (Adaptive View Retrieval vs. fine-tuned CLIP baseline):
  • IllusionMNIST: 96.0% vs. 91.80% (human: 96.69%)
  • IllusionFashionMNIST: 85.5% vs. 83.90% (human: 74.60%)
  • IllusionAnimals: 97.6% vs. 94.36% (human: 93.03%)
  • HC-Bench (SemVink protocol, Qwen2-VL-2B-Instruct backbone): 98.2% overall (100.0% objects, 96.4% text) vs. 17.0% original image, 2.7% zoom-out.
  • Error analysis: 58 errors on 1,170 test images (41 false negatives, mostly faint hate symbols; 17 false positives on benign digit look-alikes).

Limitations

  • False negatives concentrate on low-visibility hate symbols where figure-ground segmentation fails on complex naturalistic carriers (textured clothing, cluttered backgrounds, snow).
  • Deterministic classical transforms are not learned end-to-end; failure modes from imperfect Otsu segmentation propagate without adaptation.
  • HC-Bench gains are measured on a small backbone (Qwen2-VL-2B); zoom-out reportedly works well on larger models (72B), so the advantage may be model-size-dependent.
  • The template bank is closed-set; coverage depends on maintaining an up-to-date library of known hate slangs and symbols.
  • View bank hyperparameters (filter strengths, number of figure-ground variants) were not jointly optimized.

Relevance to Vision-Language Models

This paper directly quantifies a critical failure mode of frontier VLMs (GPT-4o, Gemini-2): perceptual access failure on adversarially structured images, where the harmful semantic layer is invisible to the model's default forward pass. It demonstrates that adding structured perceptual preprocessing—even with a frozen CLIP encoder—dramatically closes this gap, suggesting VLM robustness depends not only on alignment but on what visual evidence the model receives as input. The retrieve-and-calibrate design is also architecturally relevant: it shows that separating visual grounding (template retrieval) from semantic judgment (calibration head) can outperform end-to-end softmax classifiers for safety-critical tasks. Researchers tracking VLM safety, multimodal robustness, and vision-grounded reasoning will find this a concrete case study in the limits of standard VLM pipelines.