SleeperMark: Towards Robust Watermark against Fine-Tuning Text-to-image Diffusion Models

• URL: http://arxiv.org/abs/2412.04852v2
• Date: Sun, 30 Mar 2025 14:36:19 GMT
• Title: SleeperMark: Towards Robust Watermark against Fine-Tuning Text-to-image Diffusion Models
• Authors: Zilan Wang, Junfeng Guo, Jiacheng Zhu, Yiming Li, Heng Huang, Muhao Chen, Zhengzhong Tu,
• Abstract summary: SleeperMark is a framework designed to embed resilient watermarks into T2I diffusion models.<n>It guides the model to disentangle the watermark information from the semantic concepts it learns.<n>Our experiments demonstrate the effectiveness of SleeperMark across various types of diffusion models.
• Score: 77.80595722480074
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Recent advances in large-scale text-to-image (T2I) diffusion models have enabled a variety of downstream applications, including style customization, subject-driven personalization, and conditional generation. As T2I models require extensive data and computational resources for training, they constitute highly valued intellectual property (IP) for their legitimate owners, yet making them incentive targets for unauthorized fine-tuning by adversaries seeking to leverage these models for customized, usually profitable applications. Existing IP protection methods for diffusion models generally involve embedding watermark patterns and then verifying ownership through generated outputs examination, or inspecting the model's feature space. However, these techniques are inherently ineffective in practical scenarios when the watermarked model undergoes fine-tuning, and the feature space is inaccessible during verification ((i.e., black-box setting). The model is prone to forgetting the previously learned watermark knowledge when it adapts to a new task. To address this challenge, we propose SleeperMark, a novel framework designed to embed resilient watermarks into T2I diffusion models. SleeperMark explicitly guides the model to disentangle the watermark information from the semantic concepts it learns, allowing the model to retain the embedded watermark while continuing to be adapted to new downstream tasks. Our extensive experiments demonstrate the effectiveness of SleeperMark across various types of diffusion models, including latent diffusion models (e.g., Stable Diffusion) and pixel diffusion models (e.g., DeepFloyd-IF), showing robustness against downstream fine-tuning and various attacks at both the image and model levels, with minimal impact on the model's generative capability. The code is available at https://github.com/taco-group/SleeperMark.

Related papers

• AGATE: Stealthy Black-box Watermarking for Multimodal Model Copyright Protection [26.066755429896926]
  Methods select Out-of-Distribution (OoD) data as backdoor watermarks and retrain the original model for copyright protection. Existing methods are susceptible to malicious detection and forgery by adversaries, resulting in watermark evasion. We propose Model-underlineagnostic Black-box Backdoor Wunderlineatermarking Framework (AGATE) to address stealthiness and robustness challenges in multimodal model copyright protection.
  arXiv  Detail & Related papers  (2025-04-28T14:52:01Z)
• Exploiting Watermark-Based Defense Mechanisms in Text-to-Image Diffusion Models for Unauthorized Data Usage [14.985938758090763]
  Text-to-image diffusion models, such as Stable Diffusion, have shown exceptional potential in generating high-quality images.<n>Recent studies highlight concerns over the use of unauthorized data in training these models, which may lead to intellectual property infringement or privacy violations.<n>We propose RATTAN, that leverages the diffusion process to conduct controlled image generation on the protected input.
  arXiv  Detail & Related papers  (2024-11-22T22:28:19Z)
• AquaLoRA: Toward White-box Protection for Customized Stable Diffusion Models via Watermark LoRA [67.68750063537482]
  Diffusion models have achieved remarkable success in generating high-quality images. Recent works aim to let SD models output watermarked content for post-hoc forensics. We propose textttmethod as the first implementation under this scenario.
  arXiv  Detail & Related papers  (2024-05-18T01:25:47Z)
• Gaussian Shading: Provable Performance-Lossless Image Watermarking for Diffusion Models [71.13610023354967]
  Copyright protection and inappropriate content generation pose challenges for the practical implementation of diffusion models. We propose a diffusion model watermarking technique that is both performance-lossless and training-free.
  arXiv  Detail & Related papers  (2024-04-07T13:30:10Z)
• A Watermark-Conditioned Diffusion Model for IP Protection [31.969286898467985]
  We propose a unified watermarking framework for content copyright protection within the context of diffusion models. To tackle this challenge, we propose a Watermark-conditioned Diffusion model called WaDiff. Our method is effective and robust in both the detection and owner identification tasks.
  arXiv  Detail & Related papers  (2024-03-16T11:08:15Z)
• TokenMark: A Modality-Agnostic Watermark for Pre-trained Transformers [67.57928750537185]
  TokenMark is a robust, modality-agnostic, robust watermarking system for pre-trained models. It embeds the watermark by fine-tuning the pre-trained model on a set of specifically permuted data samples. It significantly improves the robustness, efficiency, and universality of model watermarking.
  arXiv  Detail & Related papers  (2024-03-09T08:54:52Z)
• Towards Robust Model Watermark via Reducing Parametric Vulnerability [57.66709830576457]
  backdoor-based ownership verification becomes popular recently, in which the model owner can watermark the model. We propose a mini-max formulation to find these watermark-removed models and recover their watermark behavior. Our method improves the robustness of the model watermarking against parametric changes and numerous watermark-removal attacks.
  arXiv  Detail & Related papers  (2023-09-09T12:46:08Z)
• Intellectual Property Protection of Diffusion Models via the Watermark Diffusion Process [22.38407658885059]
  This paper introduces WDM, a novel watermarking solution for diffusion models without imprinting the watermark during task generation. It involves training a model to concurrently learn a Watermark Diffusion Process (WDP) for embedding watermarks alongside the standard diffusion process for task generation.
  arXiv  Detail & Related papers  (2023-06-06T06:31:07Z)
• Model Watermarking for Image Processing Networks [120.918532981871]
  How to protect the intellectual property of deep models is a very important but seriously under-researched problem. We propose the first model watermarking framework for protecting image processing models.
  arXiv  Detail & Related papers  (2020-02-25T18:36:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.