Securing Self-supervised Data Curation for Foundation Models Robustness

• URL: http://arxiv.org/abs/2606.09511v1
• Date: Mon, 08 Jun 2026 14:03:51 GMT
• Title: Securing Self-supervised Data Curation for Foundation Models Robustness
• Abstract summary: Self-supervised learning (SSL) for data curation provides a pathway to scaling and improving the generalization capabilities of machine learning models.<n> SSL greatly alleviates the costs associated with annotation and manual dataset curation while minimizing the need for human oversight.<n>We propose a Poisoned Data Detector (PDD) to ensure the integrity of SSL-curated datasets prior to foundation model training.
• Score: 0.18352113484137625
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-supervised data curation provides a pathway to scaling and improving the generalization capabilities of machine learning models. By leveraging self-supervised learning (SSL) for data curation, the demand for massive training datasets required by foundation models can be effectively met. SSL greatly alleviates the costs associated with annotation and manual dataset curation while minimizing the need for human oversight. However, the integrity of SSL-curated datasets must be rigorously checked, as reliance on anonymous and unvetted external sources can substantially increase the risk of data poisoning. In this paper, we propose a Poisoned Data Detector (PDD), an active defense mechanism designed to ensure the integrity of SSL-curated datasets prior to foundation model training. PDDs are designed using a combination of the pretrained ImageBind model and traditional classifiers, including Random Forest (RF), k-Nearest Neighbors (KNN), Naive Bayes (NB), and Support Vector Machines (SVM). We rigorously evaluated PDDs using 176,200 images from three diverse datasets and three different adversarial attacks encompassing both in-distribution and out-of-distribution scenarios. Notably, SVM-PDD achieves superior performance for both in-distribution (Set3-Set5) and out-of-distribution (TrueFace and 140K RealFace) datasets. Our design demonstrates strong scalability and enables the rapid integration of new adversarial attack detectors through an ensemble approach.

Related papers

• Foundation Models for Face Presentation Attack Detection: A Unified Linear-Probing Benchmark [32.89408229880636]
  Face presentation attack detection remains challenging under cross-dataset evaluation.<n>Do general-purpose vision foundation models encode PAD-relevant information accessible with minimal task-specific training?<n>We evaluate 24 frozen encoders, including self-supervised vision transformers, vision-language encoders, and supervised CNNs.
  arXiv  Detail & Related papers  (2026-07-29T14:52:25Z)
• No Query, No Access [50.18709429731724]
  We introduce the textbfVictim Data-based Adrial Attack (VDBA), which operates using only victim texts.<n>To prevent access to the victim model, we create a shadow dataset with publicly available pre-trained models and clustering methods.<n>Experiments on the Emotion and SST5 datasets show that VDBA outperforms state-of-the-art methods, achieving an ASR improvement of 52.08%.
  arXiv  Detail & Related papers  (2025-05-12T06:19:59Z)
• Robust training of implicit generative models for multivariate and heavy-tailed distributions with an invariant statistical loss [0.5249805590164902]
  We build on the textitinvariant statistical loss (ISL) method introduced in citede2024training.<n>We extend it to handle heavy-tailed and multivariate data distributions.<n>We assess its performance in generative generative modeling and explore its potential as a pretraining technique for generative adversarial networks (GANs)
  arXiv  Detail & Related papers  (2024-10-29T10:27:50Z)
• Adversarial Robustification via Text-to-Image Diffusion Models [56.37291240867549]
  Adrial robustness has been conventionally believed as a challenging property to encode for neural networks. We develop a scalable and model-agnostic solution to achieve adversarial robustness without using any data.
  arXiv  Detail & Related papers  (2024-07-26T10:49:14Z)
• From Zero to Hero: Detecting Leaked Data through Synthetic Data Injection and Model Querying [10.919336198760808]
  We introduce a novel methodology to detect leaked data that are used to train classification models. textscLDSS involves injecting a small volume of synthetic data--characterized by local shifts in class distribution--into the owner's dataset. This enables the effective identification of models trained on leaked data through model querying alone.
  arXiv  Detail & Related papers  (2023-10-06T10:36:28Z)
• Progressive Feature Adjustment for Semi-supervised Learning from Pretrained Models [39.42802115580677]
  Semi-supervised learning (SSL) can leverage both labeled and unlabeled data to build a predictive model. Recent literature suggests that naively applying state-of-the-art SSL with a pretrained model fails to unleash the full potential of training data. We propose to use pseudo-labels from the unlabelled data to update the feature extractor that is less sensitive to incorrect labels.
  arXiv  Detail & Related papers  (2023-09-09T01:57:14Z)
• Enhancing Multiple Reliability Measures via Nuisance-extended Information Bottleneck [77.37409441129995]
  In practical scenarios where training data is limited, many predictive signals in the data can be rather from some biases in data acquisition. We consider an adversarial threat model under a mutual information constraint to cover a wider class of perturbations in training. We propose an autoencoder-based training to implement the objective, as well as practical encoder designs to facilitate the proposed hybrid discriminative-generative training.
  arXiv  Detail & Related papers  (2023-03-24T16:03:21Z)
• Open-Set Semi-Supervised Learning for 3D Point Cloud Understanding [62.17020485045456]
  It is commonly assumed in semi-supervised learning (SSL) that the unlabeled data are drawn from the same distribution as that of the labeled ones. We propose to selectively utilize unlabeled data through sample weighting, so that only conducive unlabeled data would be prioritized.
  arXiv  Detail & Related papers  (2022-05-02T16:09:17Z)
• Attentive Prototypes for Source-free Unsupervised Domain Adaptive 3D Object Detection [85.11649974840758]
  3D object detection networks tend to be biased towards the data they are trained on. We propose a single-frame approach for source-free, unsupervised domain adaptation of lidar-based 3D object detectors.
  arXiv  Detail & Related papers  (2021-11-30T18:42:42Z)
• Explaining Adversarial Vulnerability with a Data Sparsity Hypothesis [0.0]
  deep learning models are susceptible to adversarial attacks. In this paper, we develop a training framework for DL models to learn such decision boundaries. We measure adversarial robustness of the models trained using this training framework against well-known adversarial attacks.
  arXiv  Detail & Related papers  (2021-03-01T06:04:31Z)
• Learning while Respecting Privacy and Robustness to Distributional Uncertainties and Adversarial Data [66.78671826743884]
  The distributionally robust optimization framework is considered for training a parametric model. The objective is to endow the trained model with robustness against adversarially manipulated input data. Proposed algorithms offer robustness with little overhead.
  arXiv  Detail & Related papers  (2020-07-07T18:25:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.