PATFinger: Prompt-Adapted Transferable Fingerprinting against Unauthorized Multimodal Dataset Usage

• URL: http://arxiv.org/abs/2504.11509v2
• Date: Thu, 17 Apr 2025 06:10:27 GMT
• Title: PATFinger: Prompt-Adapted Transferable Fingerprinting against Unauthorized Multimodal Dataset Usage
• Authors: Wenyi Zhang, Ju Jia, Xiaojun Jia, Yihao Huang, Xinfeng Li, Cong Wu, Lina Wang,
• Abstract summary: multimodal datasets can be leveraged to pre-train vision-adapted models by providing cross-modal semantics.<n>We propose a novel prompt-language transferable fingerprinting scheme called PATFinger.<n>Our scheme utilizes inherent dataset attributes as fingerprints instead of compelling the model to learn triggers.
• Score: 19.031839603738057
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The multimodal datasets can be leveraged to pre-train large-scale vision-language models by providing cross-modal semantics. Current endeavors for determining the usage of datasets mainly focus on single-modal dataset ownership verification through intrusive methods and non-intrusive techniques, while cross-modal approaches remain under-explored. Intrusive methods can adapt to multimodal datasets but degrade model accuracy, while non-intrusive methods rely on label-driven decision boundaries that fail to guarantee stable behaviors for verification. To address these issues, we propose a novel prompt-adapted transferable fingerprinting scheme from a training-free perspective, called PATFinger, which incorporates the global optimal perturbation (GOP) and the adaptive prompts to capture dataset-specific distribution characteristics. Our scheme utilizes inherent dataset attributes as fingerprints instead of compelling the model to learn triggers. The GOP is derived from the sample distribution to maximize embedding drifts between different modalities. Subsequently, our PATFinger re-aligns the adaptive prompt with GOP samples to capture the cross-modal interactions on the carefully crafted surrogate model. This allows the dataset owner to check the usage of datasets by observing specific prediction behaviors linked to the PATFinger during retrieval queries. Extensive experiments demonstrate the effectiveness of our scheme against unauthorized multimodal dataset usage on various cross-modal retrieval architectures by 30% over state-of-the-art baselines.

Related papers

• CBW: Towards Dataset Ownership Verification for Speaker Verification via Clustering-based Backdoor Watermarking [85.68235482145091]
  Large-scale speech datasets have become valuable intellectual property.<n>We propose a novel dataset ownership verification method.<n>Our approach introduces a clustering-based backdoor watermark (CBW)<n>We conduct extensive experiments on benchmark datasets, verifying the effectiveness and robustness of our method against potential adaptive attacks.
  arXiv  Detail & Related papers  (2025-03-02T02:02:57Z)
• Transferable Unsupervised Outlier Detection Framework for Human Semantic Trajectories [9.816270572121724]
  We propose Transferable Outlier Detection for Human Semantic Trajectories (TOD4Traj) framework. ToD4Traj first introduces a modality feature unification module to align diverse data feature representations. A contrastive learning module is further pro-posed for identifying regular mobility patterns both temporally and across populations.
  arXiv  Detail & Related papers  (2024-09-28T22:31:00Z)
• Downstream-Pretext Domain Knowledge Traceback for Active Learning [138.02530777915362]
  We propose a downstream-pretext domain knowledge traceback (DOKT) method that traces the data interactions of downstream knowledge and pre-training guidance. DOKT consists of a traceback diversity indicator and a domain-based uncertainty estimator. Experiments conducted on ten datasets show that our model outperforms other state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-07-20T01:34:13Z)
• SKADA-Bench: Benchmarking Unsupervised Domain Adaptation Methods with Realistic Validation On Diverse Modalities [55.87169702896249]
  Unsupervised Domain Adaptation (DA) consists of adapting a model trained on a labeled source domain to perform well on an unlabeled target domain with some data distribution shift.<n>We present a complete and fair evaluation of existing shallow algorithms, including reweighting, mapping, and subspace alignment.<n>Our benchmark highlights the importance of realistic validation and provides practical guidance for real-life applications.
  arXiv  Detail & Related papers  (2024-07-16T12:52:29Z)
• Improving Transferability for Cross-domain Trajectory Prediction via Neural Stochastic Differential Equation [41.09061877498741]
  discrepancies exist among datasets due to external factors and data acquisition strategies. The proficient performance of models trained on large-scale datasets has limited transferability on other small-size datasets. We propose a method based on continuous and utilization of Neural Differential Equations (NSDE) for alleviating discrepancies. The effectiveness of our method is validated against state-of-the-art trajectory prediction models on the popular benchmark datasets: nuScenes, Argoverse, Lyft, INTERACTION, and Open Motion dataset.
  arXiv  Detail & Related papers  (2023-12-26T06:50:29Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• SUMMIT: Source-Free Adaptation of Uni-Modal Models to Multi-Modal Targets [30.262094419776208]
  Current approaches assume that the source data is available during adaptation and that the source consists of paired multi-modal data. We propose a switching framework which automatically chooses between two complementary methods of cross-modal pseudo-label fusion. Our method achieves an improvement in mIoU of up to 12% over competing baselines.
  arXiv  Detail & Related papers  (2023-08-23T02:57:58Z)
• The Wyner Variational Autoencoder for Unsupervised Multi-Layer Wireless Fingerprinting [6.632671046812309]
  We propose a multi-layer fingerprinting framework that jointly considers the multi-layer signatures for improved identification performance. In contrast to previous works, by leveraging the recent multi-view machine learning paradigm, our method can cluster the device information shared among the multi-layer features without supervision. Our empirical results show that the proposed method outperforms the state-of-the-art baselines in both supervised and unsupervised settings.
  arXiv  Detail & Related papers  (2023-03-28T10:05:06Z)
• Diversity inducing Information Bottleneck in Model Ensembles [73.80615604822435]
  In this paper, we target the problem of generating effective ensembles of neural networks by encouraging diversity in prediction. We explicitly optimize a diversity inducing adversarial loss for learning latent variables and thereby obtain diversity in the output predictions necessary for modeling multi-modal data. Compared to the most competitive baselines, we show significant improvements in classification accuracy, under a shift in the data distribution.
  arXiv  Detail & Related papers  (2020-03-10T03:10:41Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.