DistractMIA: Black-Box Membership Inference on Vision-Language Models via Semantic Distraction

• URL: http://arxiv.org/abs/2605.12574v1
• Date: Tue, 12 May 2026 12:04:11 GMT
• Title: DistractMIA: Black-Box Membership Inference on Vision-Language Models via Semantic Distraction
• Abstract summary: Vision-language models (VLMs) are trained on large-scale image-text corpora that may contain private, copyrighted, or otherwise sensitive data.<n>We propose DistractMIA, an output-only black-box framework based on semantic distraction.<n>DistractMIA preserves the original image, inserts a known semantic distractor, and measures how generated responses change.
• Score: 9.614855021411351
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) are trained on large-scale image-text corpora that may contain private, copyrighted, or otherwise sensitive data, motivating membership inference as a tool for training-data auditing. This is especially challenging for deployed VLMs, where auditors typically observe only generated textual responses. Existing VLM membership inference attacks either rely on probability-level signals unavailable in such settings, or use mask-based semantic prediction tasks whose effectiveness depends on object-centric visual assumptions. To address these limitations, we propose DistractMIA, an output-only black-box framework based on semantic distraction. Rather than removing visual evidence, DistractMIA preserves the original image, inserts a known semantic distractor, and measures how generated responses change. This design is motivated by the intuition that member samples remain more anchored to the original image semantics, while non-member samples are more easily redirected toward the distractor. To make this signal reliable, DistractMIA calibrates distractor configurations on a reference set and derives membership scores from repeated textual generations, capturing response stability and distractor uptake without accessing logits, probabilities, or hidden states. Experiments across multiple VLMs and benchmarks show that DistractMIA consistently outperforms both output-only and stronger-access baselines. Its performance on a medical benchmark further demonstrates applicability beyond object-centric natural images.

Related papers

• Are Reasoning Vision-Language Models Robust to Semantic Visual Distractions? [34.294060886423985]
  Reasoning Vision-Language Models (VLMs) achieve strong performance on complex multimodal tasks.<n>We introduce textbfDistract-Bench, a benchmark for evaluating VLM robustness to textbfsemantic visual distractions<n>Our results show that Distract-Bench exposes a robustness failure distinct from vision corruptions.
  arXiv  Detail & Related papers  (2026-06-08T00:40:32Z)
• Diagnosing Visual Ignorance in Vision-Language Models [29.2851901986069]
  Vision-Language Models (VLMs) frequently rely on language priors, producing confident answers that are weakly grounded in visual evidence.<n>In this work, we study language-prior reliance from both mechanistic and behavioral perspectives.<n>We find that language-prior reliance is a systematic routing failure affecting both model internals and benchmark validity.
  arXiv  Detail & Related papers  (2026-06-05T04:16:01Z)
• Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models [154.39583176906893]
  Large language models (MLs) have advanced understanding, they remain highly prone to hallucinations in scenes.<n>We argue this stems from a failure intemporal monitoring, the ability to persistently track object dynamic identities, states, and relations over time.<n>Existing benchmarks obscure this deficit relying on single final-answer queries that can often be resolved via local visual cues or statistical priors.
  arXiv  Detail & Related papers  (2026-05-09T14:32:36Z)
• DPM++: Dynamic Masked Metric Learning for Occluded Person Re-identification [32.9985650634494]
  DPM++ is a Dynamic Masked Metric Learning framework for occluded person re-identification.<n>It learns an input-adaptive masked metric that dynamically selects reliable identity subspaces for each occluded instance.<n>It consistently outperforms previous state-of-the-art methods in both holistic and occlusion scenarios.
  arXiv  Detail & Related papers  (2026-05-07T17:47:23Z)
• Membership Inference Attacks against Large Audio Language Models [50.84901010528239]
  We present the first systematic Membership Inference Attack (MIA) evaluation of Large Audio Language Models (LALMs)<n>Using a multi-modal blind baseline based on textual, spectral, and prosodic features, we demonstrate that common speech datasets exhibit near-perfect train/test separability.<n>These findings establish a principled standard for auditing LALMs beyond spurious correlations.
  arXiv  Detail & Related papers  (2026-03-30T12:45:28Z)
• Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs [47.94507630961399]
  We introduce VISAGE, a training-free decoding framework that calibrates the objective at inference time.<n>We provide an analytical stability guarantee establishing that VISAGE maintains a bounded objective loss under estimation error.<n> Evaluations across hallucination-sensitive and general-purpose benchmarks demonstrate the robustness of the framework.
  arXiv  Detail & Related papers  (2026-03-26T17:53:49Z)
• AUVIC: Adversarial Unlearning of Visual Concepts for Multi-modal Large Language Models [63.05306474002547]
  Regulatory frameworks mandating the 'right to be forgotten' drive the need for machine unlearning.<n>We introduce AUVIC, a novel visual concept unlearning framework for MLLMs.<n>We show that AUVIC achieves state-of-the-art target forgetting rates while incurs minimal performance degradation on non-target concepts.
  arXiv  Detail & Related papers  (2025-11-14T13:35:32Z)
• Context-aware TFL: A Universal Context-aware Contrastive Learning Framework for Temporal Forgery Localization [60.73623588349311]
  We propose a universal context-aware contrastive learning framework (UniCaCLF) for temporal forgery localization.<n>Our approach leverages supervised contrastive learning to discover and identify forged instants by means of anomaly detection.<n>An efficient context-aware contrastive coding is introduced to further push the limit of instant feature distinguishability between genuine and forged instants.
  arXiv  Detail & Related papers  (2025-06-10T06:40:43Z)
• PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for Multimodal Large Language Models [27.338242898495448]
  Multimodal large language models (MLLMs) have achieved remarkable success in vision-language tasks.<n>Their reliance on vast, internet-sourced data raises significant privacy and security concerns.<n>Machine unlearning (MU) has emerged as a critical technique to address these issues.<n>PEBench is a novel benchmark designed to facilitate a thorough assessment of MU in MLLMs.
  arXiv  Detail & Related papers  (2025-03-16T15:26:20Z)
• MarvelOVD: Marrying Object Recognition and Vision-Language Models for Robust Open-Vocabulary Object Detection [107.15164718585666]
  We investigate the root cause of VLMs' biased prediction under the open vocabulary detection context. Our observations lead to a simple yet effective paradigm, coded MarvelOVD, that generates significantly better training targets. Our method outperforms the other state-of-the-arts by significant margins.
  arXiv  Detail & Related papers  (2024-07-31T09:23:57Z)
• Improving Adversarial Robustness of Masked Autoencoders via Test-time Frequency-domain Prompting [133.55037976429088]
  We investigate the adversarial robustness of vision transformers equipped with BERT pretraining (e.g., BEiT, MAE) A surprising observation is that MAE has significantly worse adversarial robustness than other BERT pretraining methods. We propose a simple yet effective way to boost the adversarial robustness of MAE.
  arXiv  Detail & Related papers  (2023-08-20T16:27:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.