Skip to content

XAI-Grounded Explanation Generation for Speech Deepfake Detection with Training-Free Multimodal Large Language Models

🕒 Published (v1): 2026-06-15 02:55 UTC · Source: Arxiv · Venue: INTERSPEECH 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes XGEG, a training-free pipeline that combines traditional XAI attribution maps (IG, LIME, Saliency, SHAP) with multimodal LLMs (Qwen2.5-VL-7B and Qwen3-Omni-30B) to generate grounded, natural-language explanations for speech deepfake detection. XAI signals are interpreted visually by a vision-LLM, then synthesized into structured textual explanations by an audio-capable LLM. The approach raises Inside Accuracy by over 45 percentage points versus a pure-audio LLM baseline and produces a publicly released dataset of ~65,000 explanation instances.

Problem

Existing SDD explanation methods suffer from two complementary failures: traditional XAI (IG, LIME, SHAP, Saliency) produces low-level attribution signals that require model access and are hard for humans to interpret; LLM-based explanations are human-readable but prone to hallucination and lack grounding in acoustic evidence because no dedicated SDD explanation dataset exists. Neither approach alone provides specific, faithful, human-understandable explanations.

Method

XGEG operates in three stages without any task-specific fine-tuning: 1. XAI attribution: Three pre-trained SSL models (HuBERT, Wav2Vec 2.0, WavLM) generate spectrogram heatmaps via IG, LIME, and Saliency; a lightweight MLP classifier on eGeMAPSv02 features provides SHAP importance scores for the top acoustic features. 2. Vision-LLM interpretation: Spectrogram heatmaps are fed to Qwen2.5-VL-7B, which summarizes abnormal temporal and frequency ranges from each attribution map into structured text. 3. Multimodal synthesis: The time-frequency summaries plus top-3 SHAP features are fed to Qwen3-Omni-30B, which integrates raw audio with XAI-derived evidence to produce a structured explanation (audio abnormality time/frequency range, free-text explanation, XAI aggregation note). The dataset retains only samples correctly classified by all four models and focuses on spoofed speech; ~65k explanation instances are released.

Key Contributions

  • Training-free XAI-grounded explanation pipeline (XGEG) combining cross-model XAI aggregation with multimodal LLMs.
  • Construction and public release of a large-scale explainable SDD dataset (~65k instances) based on PartialSpoof.
  • Novel Area-Normalised Local Logit Sensitivity metric for fidelity evaluation that avoids destructive masking artifacts in time-frequency audio.
  • Empirical demonstration (human evaluation + quantitative) that XAI grounding substantially reduces hallucination and improves temporal localization precision.

Results

  • Inside Accuracy (IA): XAI-guided methods raise IA from 0.049 (pure audio baseline) to 0.482–0.811 across single-XAI conditions; single-model all-XAI reaches 0.492.
  • Area-Normalised Local Logit Sensitivity: LIME guidance achieves 46.59 ×10⁻⁶ (327× baseline); all-XAI single-model 3.15 ×10⁻⁶ (22.1× baseline); all-XAI three-model 3.12 ×10⁻⁶ (21.9× baseline).
  • Human evaluation (5-point Likert, N=600): All-XAI single-model achieves C=3.90, E=3.75, S=3.55 vs. pure-audio baseline C=3.15, E=1.75, S=1.90; three-model aggregation scores highest on specificity (S=4.30) and overall preference (O=1.50 vs. 0.35 baseline).
  • IoU is highest for pure-audio (0.269) due to over-expanded intervals, reflecting poor precision rather than good localization; XAI-guided methods sacrifice IoU for far superior IA.

Limitations

  • Cross-model XAI aggregation increases reasoning complexity for the LLM, reducing stability (lower IA than single-model full-XAI despite more evidence).
  • Bona fide speech explanation is inherently harder (absence of anomalies) and not the focus of this work.
  • No SHAP-only condition tested due to lack of explicit time-frequency localization.
  • Pre-trained models are used without fine-tuning on PartialSpoof; their moderate F1 scores (0.40–0.42) mean the XAI attribution quality is bounded by detector reliability.
  • Dataset is restricted to correctly classified samples, potentially biasing toward easier/cleaner deepfakes.
  • LLM reasoning over aggregated XAI signals remains a bottleneck; explanation-aware LLM training is left to future work.

Relevance to Vision-Language Models

The paper directly applies a vision-language model (Qwen2.5-VL-7B) as a spectrogram interpreter—treating XAI heatmaps as images and extracting structured time-frequency descriptions—demonstrating a concrete use of VLMs as intermediate reasoning modules rather than end-to-end generators. This extends the VLM grounding paradigm to audio forensics, where the visual modality encodes acoustic structure, and highlights both the utility and limits of VLMs for domain-specific evidence interpretation without fine-tuning. The work is relevant to researchers studying multimodal grounding, hallucination mitigation in LLMs, and the use of VLMs as perception components in larger tool-augmented pipelines.