Instance-Level Data-Use Auditing of Visual ML Models

• URL: http://arxiv.org/abs/2503.22413v2
• Date: Tue, 16 Sep 2025 00:34:40 GMT
• Title: Instance-Level Data-Use Auditing of Visual ML Models
• Authors: Zonghao Huang, Neil Zhenqiang Gong, Michael K. Reiter,
• Abstract summary: Growing trend of legal disputes over the unauthorized use of data in machine learning (ML) systems highlights the need for reliable data-use auditing mechanisms.<n>We present the first proactive, instance-level, data-use auditing method designed to enable data owners to audit the use of their individual data instances in ML models.
• Score: 49.862257986549885
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The growing trend of legal disputes over the unauthorized use of data in machine learning (ML) systems highlights the urgent need for reliable data-use auditing mechanisms to ensure accountability and transparency in ML. We present the first proactive, instance-level, data-use auditing method designed to enable data owners to audit the use of their individual data instances in ML models, providing more fine-grained auditing results than previous work. To do so, our research generalizes previous work integrating black-box membership inference and sequential hypothesis testing, expanding its scope of application while preserving the quantifiable and tunable false-detection rate that is its hallmark. We evaluate our method on three types of visual ML models: image classifiers, visual encoders, and vision-language models (Contrastive Language-Image Pretraining (CLIP) and Bootstrapping Language-Image Pretraining (BLIP) models). In addition, we apply our method to evaluate the performance of two state-of-the-art approximate unlearning methods. As a noteworthy second contribution, our work reveals that neither method successfully removes the influence of the unlearned data instances from image classifiers and CLIP models, even if sacrificing model utility by $10\%$.

Related papers

• LLM Unlearning Under the Microscope: A Full-Stack View on Methods and Metrics [10.638045151201084]
  We present a principled taxonomy of twelve recent stateful unlearning methods.<n>We revisit the evaluation of unlearning effectiveness (UE), utility retention (UT), and robustness (Rob)<n>Our analysis shows that current evaluations, dominated by multiple-choice question (MCQ) accuracy, offer only a narrow perspective.
  arXiv  Detail & Related papers  (2025-10-08T23:47:05Z)
• Controlling Multimodal LLMs via Reward-guided Decoding [17.5544679985101]
  We study the adaptation of Multimodal Large Language Models (MLLMs) through controlled decoding.<n>Our method involves building reward models for visual grounding and using them to guide the MLLM's decoding process.<n>We evaluate our method on standard object hallucination benchmarks, showing that it provides significant controllability over MLLM inference.
  arXiv  Detail & Related papers  (2025-08-15T17:29:06Z)
• Scalable Influence and Fact Tracing for Large Language Model Pretraining [14.598556308631018]
  Training data attribution (TDA) methods aim to attribute model outputs back to specific training examples.<n>We refine existing gradient-based methods to work effectively at scale.<n>We release our prompt set and model outputs, along with a web-based visualization tool to explore influential examples.
  arXiv  Detail & Related papers  (2024-10-22T20:39:21Z)
• Self-Comparison for Dataset-Level Membership Inference in Large (Vision-)Language Models [73.94175015918059]
  We propose a dataset-level membership inference method based on Self-Comparison. Our method does not require access to ground-truth member data or non-member data in identical distribution.
  arXiv  Detail & Related papers  (2024-10-16T23:05:59Z)
• A General Framework for Data-Use Auditing of ML Models [47.369572284751285]
  We propose a general method to audit an ML model for the use of a data-owner's data in training.<n>We show the effectiveness of our proposed framework by applying it to audit data use in two types of ML models.
  arXiv  Detail & Related papers  (2024-07-21T09:32:34Z)
• Alignment Calibration: Machine Unlearning for Contrastive Learning under Auditing [33.418062986773606]
  We first propose the framework of Machine Unlearning for Contrastive learning (MUC) and adapting existing methods. We observe that several methods are mediocre unlearners and existing auditing tools may not be sufficient for data owners to validate the unlearning effects in contrastive learning. We propose a novel method called Alignment (AC) by explicitly considering the properties of contrastive learning and optimizing towards novel metrics to easily verify unlearning.
  arXiv  Detail & Related papers  (2024-06-05T19:55:45Z)
• Improving Input-label Mapping with Demonstration Replay for In-context Learning [67.57288926736923]
  In-context learning (ICL) is an emerging capability of large autoregressive language models. We propose a novel ICL method called Sliding Causal Attention (RdSca) We show that our method significantly improves the input-label mapping in ICL demonstrations.
  arXiv  Detail & Related papers  (2023-10-30T14:29:41Z)
• Towards Better Modeling with Missing Data: A Contrastive Learning-based Visual Analytics Perspective [7.577040836988683]
  Missing data can pose a challenge for machine learning (ML) modeling. Current approaches are categorized into feature imputation and label prediction. This study proposes a Contrastive Learning framework to model observed data with missing values.
  arXiv  Detail & Related papers  (2023-09-18T13:16:24Z)
• Learn to Unlearn: A Survey on Machine Unlearning [29.077334665555316]
  This article presents a review of recent machine unlearning techniques, verification mechanisms, and potential attacks. We highlight emerging challenges and prospective research directions. We aim for this paper to provide valuable resources for integrating privacy, equity, andresilience into ML systems.
  arXiv  Detail & Related papers  (2023-05-12T14:28:02Z)
• Generalization Properties of Retrieval-based Models [50.35325326050263]
  Retrieval-based machine learning methods have enjoyed success on a wide range of problems. Despite growing literature showcasing the promise of these models, the theoretical underpinning for such models remains underexplored. We present a formal treatment of retrieval-based models to characterize their generalization ability.
  arXiv  Detail & Related papers  (2022-10-06T00:33:01Z)
• Certifiable Machine Unlearning for Linear Models [1.484852576248587]
  Machine unlearning is the task of updating machine learning (ML) models after a subset of the training data they were trained on is deleted. We present an experimental study of the three state-of-the-art approximate unlearning methods for linear models.
  arXiv  Detail & Related papers  (2021-06-29T05:05:58Z)
• Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resources [78.72922528736011]
  We propose a novel approach, black-box adversarial reprogramming (BAR), that repurposes a well-trained black-box machine learning model. Using zeroth order optimization and multi-label mapping techniques, BAR can reprogram a black-box ML model solely based on its input-output responses. BAR outperforms state-of-the-art methods and yields comparable performance to the vanilla adversarial reprogramming method.
  arXiv  Detail & Related papers  (2020-07-17T01:52:34Z)
• How Training Data Impacts Performance in Learning-based Control [67.7875109298865]
  This paper derives an analytical relationship between the density of the training data and the control performance. We formulate a quality measure for the data set, which we refer to as $rho$-gap. We show how the $rho$-gap can be applied to a feedback linearizing control law.
  arXiv  Detail & Related papers  (2020-05-25T12:13:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.