How Worrying Are Privacy Attacks Against Machine Learning?

• URL: http://arxiv.org/abs/2511.10516v1
• Date: Fri, 14 Nov 2025 01:55:54 GMT
• Title: How Worrying Are Privacy Attacks Against Machine Learning?
• Authors: Josep Domingo-Ferrer,
• Abstract summary: Given a trained model, it is necessary to mount a privacy attack to make inferences on the training data.<n>Our discussion shows that most of these attacks seem less effective in the real world than what a prima face interpretation of the related literature could suggest.
• Score: 3.871154113576347
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In several jurisdictions, the regulatory framework on the release and sharing of personal data is being extended to machine learning (ML). The implicit assumption is that disclosing a trained ML model entails a privacy risk for any personal data used in training comparable to directly releasing those data. However, given a trained model, it is necessary to mount a privacy attack to make inferences on the training data. In this concept paper, we examine the main families of privacy attacks against predictive and generative ML, including membership inference attacks (MIAs), property inference attacks, and reconstruction attacks. Our discussion shows that most of these attacks seem less effective in the real world than what a prima face interpretation of the related literature could suggest.

Related papers

• Efficient Unlearning with Privacy Guarantees [3.5502600490147196]
  We present emphefficient unlearning with privacy guarantees (EUPG)<n>EUPG offers formal privacy guarantees to individuals whose data are being unlearned.<n>Our approach demonstrates utility and forgetting effectiveness comparable to those of exact unlearning methods.
  arXiv  Detail & Related papers  (2025-07-07T08:46:02Z)
• Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models [112.48136829374741]
  In this paper, we unveil a new vulnerability: the privacy backdoor attack. When a victim fine-tunes a backdoored model, their training data will be leaked at a significantly higher rate than if they had fine-tuned a typical model. Our findings highlight a critical privacy concern within the machine learning community and call for a reevaluation of safety protocols in the use of open-source pre-trained models.
  arXiv  Detail & Related papers  (2024-04-01T16:50:54Z)
• Can Language Models be Instructed to Protect Personal Information? [30.187731765653428]
  We introduce PrivQA -- a benchmark to assess the privacy/utility trade-off when a model is instructed to protect specific categories of personal information in a simulated scenario. We find that adversaries can easily circumvent these protections with simple jailbreaking methods through textual and/or image inputs. We believe PrivQA has the potential to support the development of new models with improved privacy protections, as well as the adversarial robustness of these protections.
  arXiv  Detail & Related papers  (2023-10-03T17:30:33Z)
• Students Parrot Their Teachers: Membership Inference on Model Distillation [54.392069096234074]
  We study the privacy provided by knowledge distillation to both the teacher and student training sets. Our attacks are strongest when student and teacher sets are similar, or when the attacker can poison the teacher set.
  arXiv  Detail & Related papers  (2023-03-06T19:16:23Z)
• Membership Inference Attacks against Synthetic Data through Overfitting Detection [84.02632160692995]
  We argue for a realistic MIA setting that assumes the attacker has some knowledge of the underlying data distribution. We propose DOMIAS, a density-based MIA model that aims to infer membership by targeting local overfitting of the generative model.
  arXiv  Detail & Related papers  (2023-02-24T11:27:39Z)
• Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining [75.25943383604266]
  We question whether the use of large Web-scraped datasets should be viewed as differential-privacy-preserving. We caution that publicizing these models pretrained on Web data as "private" could lead to harm and erode the public's trust in differential privacy as a meaningful definition of privacy. We conclude by discussing potential paths forward for the field of private learning, as public pretraining becomes more popular and powerful.
  arXiv  Detail & Related papers  (2022-12-13T10:41:12Z)
• The Privacy Onion Effect: Memorization is Relative [76.46529413546725]
  We show an Onion Effect of memorization: removing the "layer" of outlier points that are most vulnerable exposes a new layer of previously-safe points to the same attack. It suggests that privacy-enhancing technologies such as machine unlearning could actually harm the privacy of other users.
  arXiv  Detail & Related papers  (2022-06-21T15:25:56Z)
• Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets [53.866927712193416]
  We show that an adversary who can poison a training dataset can cause models trained on this dataset to leak private details belonging to other parties. Our attacks are effective across membership inference, attribute inference, and data extraction. Our results cast doubts on the relevance of cryptographic privacy guarantees in multiparty protocols for machine learning.
  arXiv  Detail & Related papers  (2022-03-31T18:06:28Z)
• Investigating Membership Inference Attacks under Data Dependencies [26.70764798408236]
  Training machine learning models on privacy-sensitive data has opened the door to new attacks that can have serious privacy implications. One such attack, the Membership Inference Attack (MIA), exposes whether or not a particular data point was used to train a model. We evaluate the defence under the restrictive assumption that all members of the training set, as well as non-members, are independent and identically distributed.
  arXiv  Detail & Related papers  (2020-10-23T00:16:46Z)
• When Machine Unlearning Jeopardizes Privacy [25.167214892258567]
  We investigate the unintended information leakage caused by machine unlearning. We propose a novel membership inference attack that achieves strong performance. Our results can help improve privacy protection in practical implementations of machine unlearning.
  arXiv  Detail & Related papers  (2020-05-05T14:11:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.