Multi-concept Model Immunization through Differentiable Model Merging

• URL: http://arxiv.org/abs/2412.15320v1
• Date: Thu, 19 Dec 2024 18:59:05 GMT
• Title: Multi-concept Model Immunization through Differentiable Model Merging
• Authors: Amber Yijia Zheng, Raymond A. Yeh,
• Abstract summary: Model immunization aims to mitigate the potential risk of misuse associated with open-sourced models.<n>Recent work on model immunization focuses on the single-concept setting.<n>We propose an immunization algorithm that learns a single difficult initialization'' for adaptation methods over a set of concepts.<n>We achieve this by incorporating a differentiable merging layer that combines a set of model weights adapted over multiple concepts.
• Score: 11.912092139018885
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Model immunization is an emerging direction that aims to mitigate the potential risk of misuse associated with open-sourced models and advancing adaptation methods. The idea is to make the released models' weights difficult to fine-tune on certain harmful applications, hence the name ``immunized''. Recent work on model immunization focuses on the single-concept setting. However, models need to be immunized against multiple concepts in real-world situations. To address this gap, we propose an immunization algorithm that, simultaneously, learns a single ``difficult initialization'' for adaptation methods over a set of concepts. We achieve this by incorporating a differentiable merging layer that combines a set of model weights adapted over multiple concepts. In our experiments, we demonstrate the effectiveness of multi-concept immunization by generalizing prior work's experiment setup of re-learning and personalization adaptation to multiple concepts.

Related papers

• On Arbitrary Predictions from Equally Valid Models [49.56463611078044]
  Model multiplicity refers to multiple machine learning models that admit conflicting predictions for the same patient.<n>We show that even small ensembles can mitigate/eliminate predictive multiplicity in practice.
  arXiv  Detail & Related papers  (2025-07-25T16:15:59Z)
• Model Immunization from a Condition Number Perspective [14.84123611635938]
  We propose a framework, based on the condition number of a Hessian matrix, to analyze model immunization for linear models.<n>We design an algorithm with regularization terms to control the resulting condition numbers after pre-training.<n> Empirical results on linear models and non-linear deep-nets demonstrate the effectiveness of the proposed algorithm.
  arXiv  Detail & Related papers  (2025-05-29T17:59:48Z)
• When Are Concepts Erased From Diffusion Models? [44.89615668122767]
  Concept erasure is the ability to selectively prevent a model from generating specific concepts.<n>We propose two conceptual models for the erasure mechanism in diffusion models.<n>To thoroughly assess whether a concept has been truly erased from the model, we introduce a suite of independent evaluations.
  arXiv  Detail & Related papers  (2025-05-22T17:59:09Z)
• Modular Customization of Diffusion Models via Blockwise-Parameterized Low-Rank Adaptation [73.16975077770765]
  Modular customization is essential for applications like concept stylization and multi-concept customization. Instant merging methods often cause identity loss and interference of individual merged concepts. We propose BlockLoRA, an instant merging method designed to efficiently combine multiple concepts while accurately preserving individual concepts' identity.
  arXiv  Detail & Related papers  (2025-03-11T16:10:36Z)
• Concept Complement Bottleneck Model for Interpretable Medical Image Diagnosis [8.252227380729188]
  We propose a concept complement bottleneck model for interpretable medical image diagnosis. We propose to use concept adapters for specific concepts to mine the concept differences and score concepts in their own attention channels. Our model outperforms the state-of-the-art competitors in concept detection and disease diagnosis tasks.
  arXiv  Detail & Related papers  (2024-10-20T16:52:09Z)
• Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient [20.698305103879232]
  We propose a novel concept domain correction framework named textbfDoCo (textbfDomaintextbfCorrection)<n>By aligning the output domains of sensitive and anchor concepts through adversarial training, our approach ensures comprehensive unlearning of target concepts.<n>We also introduce a concept-preserving gradient surgery technique that mitigates conflicting gradient components, thereby preserving the model's utility while unlearning specific concepts.
  arXiv  Detail & Related papers  (2024-05-24T07:47:36Z)
• Separable Multi-Concept Erasure from Diffusion Models [52.51972530398691]
  We propose a Separable Multi-concept Eraser (SepME) to eliminate unsafe concepts from large-scale diffusion models. The latter separates optimizable model weights, making each weight increment correspond to a specific concept erasure. Extensive experiments indicate the efficacy of our approach in eliminating concepts, preserving model performance, and offering flexibility in the erasure or recovery of various concepts.
  arXiv  Detail & Related papers  (2024-02-03T11:10:57Z)
• Orthogonal Adaptation for Modular Customization of Diffusion Models [39.62438974450659]
  We address a new problem called Modular Customization, with the goal of efficiently merging customized models.<n>We introduce Orthogonal Adaptation, a method designed to encourage the customized models, which do not have access to each other during fine-tuning.<n>Our proposed method is both simple and versatile, applicable to nearly all optimizable weights in the model architecture.
  arXiv  Detail & Related papers  (2023-12-05T02:17:48Z)
• MAST: Model-Agnostic Sparsified Training [4.962431253126472]
  We introduce a novel optimization problem formulation that departs from the conventional way of minimizing machine learning model loss as a black-box function. Unlike traditional formulations, the proposed approach explicitly incorporates an initially pre-trained model and random sketch operators. We present several variants of the Gradient Descent (SGD) method adapted to the new problem formulation.
  arXiv  Detail & Related papers  (2023-11-27T18:56:03Z)
• Phasic Content Fusing Diffusion Model with Directional Distribution Consistency for Few-Shot Model Adaption [73.98706049140098]
  We propose a novel phasic content fusing few-shot diffusion model with directional distribution consistency loss. Specifically, we design a phasic training strategy with phasic content fusion to help our model learn content and style information when t is large. Finally, we propose a cross-domain structure guidance strategy that enhances structure consistency during domain adaptation.
  arXiv  Detail & Related papers  (2023-09-07T14:14:11Z)
• Multi-modal Latent Diffusion [8.316365279740188]
  Multi-modal Variational Autoencoders are a popular family of models that aim to learn a joint representation of the different modalities. Existing approaches suffer from a coherence-quality tradeoff, where models with good generation quality lack generative coherence across modalities. We propose a novel method that uses a set of independently trained, uni-modal, deterministic autoencoders.
  arXiv  Detail & Related papers  (2023-06-07T14:16:44Z)
• Raising the Cost of Malicious AI-Powered Image Editing [82.71990330465115]
  We present an approach to mitigating the risks of malicious image editing posed by large diffusion models. The key idea is to immunize images so as to make them resistant to manipulation by these models.
  arXiv  Detail & Related papers  (2023-02-13T18:38:42Z)
• A Twin Neural Model for Uplift [59.38563723706796]
  Uplift is a particular case of conditional treatment effect modeling. We propose a new loss function defined by leveraging a connection with the Bayesian interpretation of the relative risk. We show our proposed method is competitive with the state-of-the-art in simulation setting and on real data from large scale randomized experiments.
  arXiv  Detail & Related papers  (2021-05-11T16:02:39Z)
• Adversarial Sample Enhanced Domain Adaptation: A Case Study on Predictive Modeling with Electronic Health Records [57.75125067744978]
  We propose a data augmentation method to facilitate domain adaptation. adversarially generated samples are used during domain adaptation. Results confirm the effectiveness of our method and the generality on different tasks.
  arXiv  Detail & Related papers  (2021-01-13T03:20:20Z)
• An Optimal Control Approach to Learning in SIDARTHE Epidemic model [67.22168759751541]
  We propose a general approach for learning time-variant parameters of dynamic compartmental models from epidemic data. We forecast the epidemic evolution in Italy and France.
  arXiv  Detail & Related papers  (2020-10-28T10:58:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.