Skip to content

Adversarial Diffusion Across Modalities: A Fusion Survey of Attacks, Defenses, and Evaluation for Text, Vision, and Vision-Language Models

🕒 Published (v1): 2026-06-25 03:32 UTC · Source: Arxiv · link

Why this paper was selected

Comprehensive survey unifying adversarial attacks across text/vision/VLMs; useful robustness reference

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

A narrative survey that unifies four previously siloed research tracks—diffusion-based attacks on LLMs, image classifiers, and VLMs, plus diffusion-based defenses—into a single conceptual framework. It catalogs 50 papers, proposes a six-class taxonomy of diffusion roles in adversarial pipelines, and introduces a five-dimension evaluation framework applied uniformly across modalities. The LLM-side cluster is the smallest (4 papers) but the fastest-growing and most methodologically diverse.

Problem

Adversarial evaluation of AI systems has fragmented across four disconnected communities (NLP safety, vision adversarial robustness, multimodal LLM safety, generative modeling), each with incompatible vocabularies, threat models, and benchmarks. Existing red teaming surveys mention diffusion-based text attacks in at most a sentence; diffusion surveys ignore adversarial use. No consolidated reference describes what has been tried, what works, and where the gaps are for diffusion-as-adversarial-generator, especially against LLMs.

Method

Structured narrative review (explicitly not PRISMA-compliant). Searched arXiv, Google Scholar, OpenReview, ACL Anthology, IEEE Xplore, ACM DL, and major ML/CV/NLP/security venue proceedings using keyword combinations of "diffusion model"/"score-based" with "adversarial"/"jailbreak"/"red teaming"/"attack," followed by recursive citation traversal. Pipeline: 154 candidates → 94 (title/abstract) → 50 retained (full-text). Inclusion criterion: diffusion must be a constitutive adversarial component (generator, victim, or explicit denoising defense). Quality assessed via an 8-item checklist (diffusion role clarity, formulation statement, metric reporting, baseline comparisons, etc.). Post-hoc inter-rater agreement estimated using two independent LLM agents with contrasting prompting strategies (strict-literal vs. reasoning-first); Cohen's \(\kappa = 0.781\) (substantial agreement, observed agreement 89.6%).

Two primary contributions:

Six-class taxonomy of diffusion roles: 1. Trained generator 2. Frozen with latent perturbation 3. Frozen with score/classifier guidance 4. Off-the-shelf inference 5. Pipeline-only renderer 6. Victim diffusion model

Augmented by a threat-model axis: attacker knowledge Ă— query budget Ă— target accessibility.

Five-dimension evaluation framework: attack success rate (ASR), transferability, query budget, perplexity, defense-evasion—applied uniformly across text, image-classifier, and VLM attacks.

Key Contributions

  • Six-class diffusion-role taxonomy plus threat-model axis, enabling cross-modality comparison of attack architectures.
  • Unified five-dimension evaluation framework exposing the "closed-frontier-model gap" that diffusion-based attacks have not yet closed.
  • Catalog of 50 papers across four streams: 4 text/LLM attacks, 18 image-classifier attacks, 10 VLM jailbreak attacks, 4 dLLM-as-victim papers, 4 diffusion-based defenses, 10 non-diffusion baselines. All metadata released as a relational spreadsheet.
  • Dual attacker-defender analysis: five recurring weaknesses of current LLM-side attack literature identified and substantiated quantitatively.
  • Research agenda of open questions with concrete experimental designs, grouped by methodological scope.

Results

  • 50 papers cataloged; 22/50 (44%) provide public code releases.
  • LLM-side attack cluster: 4 papers (DiffusionAttacker, DART, Qiu et al., LĂĽdke et al.), covering four distinct attack philosophies (continuous-latent trained generator, embedding-space \(\epsilon\)-ball diffusion, off-the-shelf dLLM inference, dual-objective training).
  • VLM jailbreak cluster: 10 papers (HADES, MM-SafetyBench, IDEATOR, Visual-RolePlay, AttackVLM, InstructTA, AdvDiffVLM, Xu et al., AnyAttack, AdvI2I), spanning score-guided, pipeline/renderer, diffusion-style pretrain, and diffusion-as-victim roles.
  • Inter-agent agreement on 154 records: \(\kappa = 0.781\); agents agreed with authors on all 50 inclusions and 88/104 exclusions; 16 boundary cases identified.
  • Image-domain cluster (18 papers) is established (accumulating from 2023); text/LLM and dLLM-victim families emerged entirely in 2024–2025.
  • Five recurring weaknesses of LLM-side literature identified (specific weaknesses not enumerated in the available text excerpt beyond the abstract-level claim).

Limitations

  • Narrative review only; protocol not pre-registered; no independent dual-author blinded full-text screening with formal inter-rater agreement statistics.
  • Post-hoc LLM-agent agreement check is a transparency probe, not a substitute for human inter-rater reliability.
  • LLM-side corpus is extremely small (4 papers), making quantitative meta-analytic conclusions premature.
  • Fast publication pace means relevant arXiv preprints may have appeared after the May 2026 search cutoff.
  • Defenses for image models (DiffPure and follow-ups) are referenced but not enumerated, creating an asymmetry in defense coverage relative to attack coverage.
  • Scope boundary decisions (e.g., exclusion of DiffCAP, patch-defense variants) involve acknowledged degrees of freedom not resolvable by the inclusion rule alone.

Relevance to Vision-Language Models

The survey devotes an entire sub-corpus to VLM adversarial attacks (10 cataloged papers), covering how diffusion-generated adversarial images penetrate multimodal systems via score-guided perturbation (AdvDiffVLM), cross-attention manipulation, and pipeline-renderer approaches (HADES, MM-SafetyBench). The unified five-dimension evaluation framework and six-class taxonomy are directly applicable to VLM safety benchmarking, providing a structured way to compare VLM jailbreak methods against non-diffusion baselines. For VLM researchers, the survey's identification of the "closed-frontier-model gap" (diffusion attacks not yet demonstrated against the strongest closed-source VLMs) marks a concrete open problem, and the image-domain recipes (DDIM-inversion plus latent perturbation; classifier-score guidance) being ported to multimodal settings signal the near-term direction of adversarial pressure on VLMs.