LOMIA: Label-Only Membership Inference Attacks against Pre-trained Large Vision-Language Models¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

LOMIA is the first label-only membership inference attack (MIA) framework targeting the pre-training phase of large vision-language models (VLLMs), operating under the restrictive assumption that only top-1 text predictions are observable. It approximates output perplexity via surrogate-model regression on semantic similarity features, achieving performance competitive with grey-box logits-based attacks on three open-source VLLMs and GPT-4o.

Problem¶

Existing MIAs against VLLMs require access to full model logits (token-level probabilities) to compute perplexity, entropy, or Rényi divergence membership signals. In real-world deployments, API access is typically limited to the final generated text (top-1 label), rendering these methods inapplicable. No prior work had demonstrated effective MIAs against pre-trained VLLMs under a strict label-only, no-shadow-model threat model.

Method¶

LOMIA introduces a two-stage framework applied to three attack variants:

Regression stage: Query a surrogate VLLM (LLaVA-1.5-7B in experiments) with the target image to obtain a generated description. Compute semantic similarity features between the description and the ground-truth caption. Fit a linear regression model mapping these features to the perplexity (PPL) computed from the surrogate's logits.

Inference stage: Query the target model (logits unavailable) to generate a description. Extract the same features and pass them through the regression model to predict a PPL proxy, which serves as the membership score (higher predicted PPL → non-member).

Three attack variants extract different feature pairs for regression: - TTFA (text-text): Text-text cosine similarity (all-MiniLM-L6-v2 sentence embeddings) → predicted PPL. - ITFA (image-text): CLIP image-text cosine similarity between generated description and original image → predicted PPL. - DUFA (dual): Both text-text and image-text similarities jointly → predicted PPL via multivariate linear regression.

The core intuition: member samples are more memorized, producing higher cross-modal alignment and lower perplexity in generated outputs.

Key Contributions¶

First label-only MIA framework targeting the pre-training phase of VLLMs, requiring no logit access and no shadow model.
Three complementary attack methods (TTFA, ITFA, DUFA) exploiting text-text similarity, image-text alignment, and their combination as proxy signals for perplexity.
A new multimodal MIA evaluation dataset (LOMIA/LAION and LOMIA/CC) with 600 image-text pairs each (300 members, 300 non-members), using Stable Diffusion to generate non-member images from member captions.
Demonstration that label-only attacks can match grey-box logits-based attacks and extend to closed-source GPT-4o.

Results¶

vs. label-only baseline (Query Attack): TTFA improves AUC by 0.085 (LAION: 0.516→0.601) and 0.069 (CC: 0.599→0.584 vs 0.625 for ITFA) on LLaVA-1.5; TPR@1%FPR improves from 0.3% to up to 4.6% (ITFA on CC/LLaVA).
vs. logits-based attacks: LOMIA AUC (0.601–0.630) approaches top logits-based baselines (e.g., Max10%Rényi α=2: AUC 0.750/0.891 on LLaVA/MiniGPT-4) but falls short on TPR@1%FPR (LOMIA ≤4.6% vs. Max0%Rényi α=1 at 13% on LLaMA-Adapter/LAION).
GPT-4o: ITFA achieves AUC 0.669 (LAION), DUFA achieves AUC 0.618 (CC) with balanced accuracy 0.600.
DUFA robustness: AUC varies only 0.618–0.621 across four sentence embedding models; outperforms single-feature variants on larger, real-image datasets.
Architecture sensitivity: DUFA performs best when surrogate and target share architecture; TTFA degrades substantially on real-image expanded datasets where DUFA is most robust.

Limitations¶

AUC values are modest (0.57–0.63 for label-only variants), significantly below the best logits-based methods on TPR@1%FPR, the most practically relevant metric for auditing.
Performance degrades when surrogate and target model architectures differ, limiting generalizability in black-box commercial settings.
Membership signal assumes the attacker has access to the ground-truth captions/labels of the queried images, which may not hold in practice.
Evaluation dataset size is small (600 samples per split), and non-member images are synthetically generated via Stable Diffusion from member captions—a potentially confounded construction that may inflate or deflate attack signals.
The linear regression between similarity and PPL is a simplification; non-linear interactions are unmodeled.
No defense analysis or evaluation of LOMIA against privacy-preserving training mechanisms (e.g., DP-SGD).

Relevance to Vision-Language Models¶

This paper directly targets the privacy risks inherent in the large-scale internet-scraped pretraining datasets that underpin modern VLLMs (LLaVA, MiniGPT-4, LLaMA-Adapter, GPT-4o). It demonstrates that the memorization behavior of VLLMs—exploited through cross-modal semantic alignment in CLIP embedding space—constitutes a measurable and practical attack surface even without logit access. For VLM researchers, this work motivates privacy-aware pretraining practices and highlights that multimodal memorization is a distinct and underexplored axis of vulnerability compared to unimodal LLMs. The surrogate-based regression paradigm for approximating internal model statistics from label-only outputs is broadly applicable to other closed-API multimodal systems.