Skip to content

InfraQR: Edge-Placed QR-Inspired Structured Patch Attacks on Infrared Vision-Language Models

🕒 Published (v1): 2026-07-08 11:28 UTC · Source: Arxiv · link

Why this paper was selected

Structured adversarial patch attacks on infrared VLMs; underexplored robustness axis with security relevance

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

InfraQR proposes a structured adversarial patch attack targeting infrared vision-language models (VLMs), borrowing QR code layout principles to construct a compact, near-binary edge-placed perturbation. Unlike prior work that attaches patches to target objects, InfraQR places a \(37\times37\) patch along image boundaries, yet consistently degrades CLIP-style classification, black-box captioning, and VQA. The attack reduces OpenAI CLIP accuracy from 98.67% to 0.70% on a 300-image infrared benchmark.

Problem

Infrared VLMs are deployed in safety-critical low-light and adverse-weather perception tasks, but adversarial robustness studies have focused almost exclusively on RGB imagery, full-image perturbations, or thermal object detectors with patches placed on target objects. It is unknown whether a small, peripheral, structured patch—placed away from the main object—can corrupt infrared vision-language understanding across classification, captioning, and VQA simultaneously.

Method

InfraQR parameterizes a \(G\times G\) (\(G=21\)) logical grid with three fixed \(7\times7\) finder-style QR anchors at top-left, top-right, and bottom-left corners. The remaining learnable cells are continuous logits \(Z\) mapped through a temperature-controlled sigmoid \(S = \sigma(\tau Z)\), \(\tau=15\), to form:

\[P^{\mathrm{log}} = F \odot A + (1-F) \odot S\]

A binary regularization term \(\mathcal{L}_{\mathrm{bin}} = \frac{1}{|\Omega|}\sum_{i\in\Omega} P^{\mathrm{log}}_i(1 - P^{\mathrm{log}}_i)\) encourages near-binary values without requiring a scannable QR code.

Placement: A constant gray probe (\(\rho=0.5\)) is tested at 16 unique boundary candidate locations; the position minimizing ground-truth CLIP similarity is selected once and frozen.

Classification loss: A ranking loss \(\mathcal{L}_{\mathrm{cls}} = \max(0,\, \tilde{s}_y^{\mathrm{adv}} - s_{\mathrm{comp}}^{\mathrm{adv}})\) drives the correct-class similarity below all competitors. Total loss: \(\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{cls}} + \lambda_{\mathrm{bin}}\mathcal{L}_{\mathrm{bin}}\).

QA-aware extension: For VQA, a template \(\phi(q, a^+, r)\) converts question/answer/type into a CLIP text target; the loss \(\mathcal{L}_{\mathrm{qa}} = s^+(x^{\mathrm{adv}}) - s^-(x^{\mathrm{adv}})\) suppresses the correct target while promoting the strongest incorrect candidate, with no access to generative VQA models during optimization.

Patches are optimized for 1000 Adam iterations (lr 0.03) on surrogate CLIP-style encoders and transferred to black-box captioning and VQA models.

Key Contributions

  • InfraQR: a QR-inspired structured adversarial patch framework combining fixed finder anchors, learnable binary-regularized grid cells, and edge-based placement for infrared VLMs.
  • Demonstration that edge-placed patches away from the target object are sufficient to strongly degrade infrared vision-language representations.
  • A multi-task evaluation protocol covering surrogate-model classification, black-box caption transfer (BLIP-2, InstructBLIP, LLaVA-1.5/1.6, OpenFlamingo), and question–answer-aware VQA attack.
  • A question-type-conditioned QA-aware loss extension that transfers to black-box generative VQA without accessing those models during optimization.

Results

  • Classification (white-box surrogates):
  • OpenAI CLIP ViT-L/14: accuracy 98.67% → 0.70% (drop of 97.97 pp); baselines HCB/AdvIC/AdvGrid drop at most 11.82 pp.
  • MetaCLIP ViT-L/14: accuracy 96.67% → 10.81% (drop 85.86 pp).
  • EVA-CLIP ViT-G/14: accuracy 98.33% → 62.40% (drop 35.93 pp).
  • OpenCLIP ViT-B/16: accuracy 96.00% → 3.64% (drop 92.36 pp).
  • Caption transfer (black-box): Adversarial images cause semantic degradation in captions across BLIP-2, InstructBLIP, LLaVA-1.5/1.6, and OpenFlamingo, measured by GPT-5.4-based semantic consistency scoring; InfraQR achieves lower consistency than all three baselines.
  • VQA (black-box): Attacked images produce more erroneous answers under GPT-5.4 evaluation on 236 test pairs (yes/no, counting, object-category, spatial-location), outperforming baselines.
  • All baselines (HCB, AdvIC, AdvGrid) use the same edge-placement locations for controlled comparison.

Limitations

  • Evaluation is limited to a 300-image benchmark from one infrared dataset (Infrared-Image-Instruct-12K); generalization to other infrared datasets and distributions is not demonstrated.
  • The attack is digital-only; physical realizability of a near-binary infrared patch on a real surface is not evaluated.
  • EVA-CLIP ViT-G/14 retains 62.40% accuracy post-attack, indicating the method is significantly weaker against larger/stronger encoders.
  • The QA-aware VQA extension relies on having a known reference answer and does not handle open-ended questions reliably.
  • No defenses are evaluated; robustness of InfraQR against detection or adversarial training is not assessed.
  • GPT-5.4 as caption/VQA judge introduces potential evaluation noise not separately validated.

Relevance to Vision-Language Models

InfraQR directly probes an underexplored vulnerability of CLIP-style VLMs when applied to infrared modalities: a small peripheral structured patch suffices to collapse zero-shot classification and degrade cross-model captioning and VQA, revealing that VLM representations are sensitive to structured edge-region interference beyond the semantic content of the scene. The surrogate-to-black-box transfer across heterogeneous VLMs (BLIP-2, InstructBLIP, LLaVA, OpenFlamingo) highlights a concerning generalization of adversarial patches in the shared vision-language embedding space. For researchers tracking VLMs, this work motivates robustness evaluation across non-RGB modalities and structured localized perturbations, complementing existing work on typographic attacks and illumination-based adversarial examples. The QA-aware loss construction—using CLIP as a surrogate to attack generative VQA models without access—is a transferable technique applicable to RGB VLM attack research.