Hufu: A Modality-Agnositc Watermarking System for Pre-Trained Transformers via Permutation Equivariance

• URL: http://arxiv.org/abs/2403.05842v1
• Date: Sat, 9 Mar 2024 08:54:52 GMT
• Title: Hufu: A Modality-Agnositc Watermarking System for Pre-Trained Transformers via Permutation Equivariance
• Authors: Hengyuan Xu, Liyao Xiang, Xingjun Ma, Borui Yang, Baochun Li
• Abstract summary: Hufu is a modality-agnostic watermarking system for pre-trained Transformer-based models. It embeds watermark by fine-tuning the pre-trained model on a set of data samples specifically permuted. It is naturally modality-agnostic, task-independent, and trigger-sample-free.
• Score: 47.35106847363781
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the blossom of deep learning models and services, it has become an imperative concern to safeguard the valuable model parameters from being stolen. Watermarking is considered an important tool for ownership verification. However, current watermarking schemes are customized for different models and tasks, hard to be integrated as an integrated intellectual protection service. We propose Hufu, a modality-agnostic watermarking system for pre-trained Transformer-based models, relying on the permutation equivariance property of Transformers. Hufu embeds watermark by fine-tuning the pre-trained model on a set of data samples specifically permuted, and the embedded model essentially contains two sets of weights -- one for normal use and the other for watermark extraction which is triggered on permuted inputs. The permutation equivariance ensures minimal interference between these two sets of model weights and thus high fidelity on downstream tasks. Since our method only depends on the model itself, it is naturally modality-agnostic, task-independent, and trigger-sample-free. Extensive experiments on the state-of-the-art vision Transformers, BERT, and GPT2 have demonstrated Hufu's superiority in meeting watermarking requirements including effectiveness, efficiency, fidelity, and robustness, showing its great potential to be deployed as a uniform ownership verification service for various Transformers.

Related papers

• Task-Agnostic Language Model Watermarking via High Entropy Passthrough Layers [11.089926858383476]
  We propose model watermarking via passthrough layers, which are added to existing pre-trained networks. Our method is fully task-agnostic, and can be applied to both classification and sequence-to-sequence tasks. We show our method is robust to both downstream fine-tuning, fine-pruning, and layer removal attacks.
  arXiv  Detail & Related papers  (2024-12-17T05:46:50Z)
• SleeperMark: Towards Robust Watermark against Fine-Tuning Text-to-image Diffusion Models [77.80595722480074]
  SleeperMark is a novel framework designed to embed resilient watermarks into T2I diffusion models. It guides the model to disentangle the watermark information from the semantic concepts it learns, allowing the model to retain the embedded watermark. Our experiments demonstrate the effectiveness of SleeperMark across various types of diffusion models.
  arXiv  Detail & Related papers  (2024-12-06T08:44:18Z)
• Towards Effective User Attribution for Latent Diffusion Models via Watermark-Informed Blending [54.26862913139299]
  We introduce a novel framework Towards Effective user Attribution for latent diffusion models via Watermark-Informed Blending (TEAWIB) TEAWIB incorporates a unique ready-to-use configuration approach that allows seamless integration of user-specific watermarks into generative models. Experiments validate the effectiveness of TEAWIB, showcasing the state-of-the-art performance in perceptual quality and attribution accuracy.
  arXiv  Detail & Related papers  (2024-09-17T07:52:09Z)
• Open-Set Deepfake Detection: A Parameter-Efficient Adaptation Method with Forgery Style Mixture [58.60915132222421]
  We introduce an approach that is both general and parameter-efficient for face forgery detection. We design a forgery-style mixture formulation that augments the diversity of forgery source domains. We show that the designed model achieves state-of-the-art generalizability with significantly reduced trainable parameters.
  arXiv  Detail & Related papers  (2024-08-23T01:53:36Z)
• EMR-Merging: Tuning-Free High-Performance Model Merging [55.03509900949149]
  We show that Elect, Mask & Rescale-Merging (EMR-Merging) shows outstanding performance compared to existing merging methods. EMR-Merging is tuning-free, thus requiring no data availability or any additional training while showing impressive performance.
  arXiv  Detail & Related papers  (2024-05-23T05:25:45Z)
• Provable Adversarial Robustness for Group Equivariant Tasks: Graphs, Point Clouds, Molecules, and More [9.931513542441612]
  We propose a sound notion of adversarial robustness that accounts for task equivariance. certification methods are, however, unavailable for many models. We derive the first architecture-specific graph edit distance certificates, i.e. sound robustness guarantees for isomorphism equivariant tasks like node classification.
  arXiv  Detail & Related papers  (2023-12-05T12:09:45Z)
• Functional Invariants to Watermark Large Transformers [30.598259061227594]
  The rapid growth of transformer-based models increases the concerns about their integrity and ownership insurance. Watermarking addresses this issue by embedding a unique identifier into the model, while preserving its performance. This paper explores watermarks with virtually no computational cost, applicable to a non-blind white-box setting.
  arXiv  Detail & Related papers  (2023-10-17T17:56:18Z)
• Certifying Model Accuracy under Distribution Shifts [151.67113334248464]
  We present provable robustness guarantees on the accuracy of a model under bounded Wasserstein shifts of the data distribution. We show that a simple procedure that randomizes the input of the model within a transformation space is provably robust to distributional shifts under the transformation.
  arXiv  Detail & Related papers  (2022-01-28T22:03:50Z)
• Characterizing and Taming Model Instability Across Edge Devices [4.592454933053539]
  This paper presents the first methodical characterization of the variations in model prediction across real-world mobile devices. We introduce a new metric, instability, which captures this variation. In experiments, 14-17% of images produced divergent classifications across one or more phone models.
  arXiv  Detail & Related papers  (2020-10-18T16:52:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.