ModelDiff: Testing-Based DNN Similarity Comparison for Model Reuse Detection

• URL: http://arxiv.org/abs/2106.08890v1
• Date: Fri, 11 Jun 2021 15:16:18 GMT
• Title: ModelDiff: Testing-Based DNN Similarity Comparison for Model Reuse Detection
• Authors: Yuanchun Li, Ziqi Zhang, Bingyan Liu, Ziyue Yang, and Yunxin Liu
• Abstract summary: ModelDiff is a testing-based approach to deep learning model similarity comparison. A study on mobile deep learning apps has shown the feasibility of ModelDiff on real-world models.
• Score: 9.106864924968251
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The knowledge of a deep learning model may be transferred to a student model, leading to intellectual property infringement or vulnerability propagation. Detecting such knowledge reuse is nontrivial because the suspect models may not be white-box accessible and/or may serve different tasks. In this paper, we propose ModelDiff, a testing-based approach to deep learning model similarity comparison. Instead of directly comparing the weights, activations, or outputs of two models, we compare their behavioral patterns on the same set of test inputs. Specifically, the behavioral pattern of a model is represented as a decision distance vector (DDV), in which each element is the distance between the model's reactions to a pair of inputs. The knowledge similarity between two models is measured with the cosine similarity between their DDVs. To evaluate ModelDiff, we created a benchmark that contains 144 pairs of models that cover most popular model reuse methods, including transfer learning, model compression, and model stealing. Our method achieved 91.7% correctness on the benchmark, which demonstrates the effectiveness of using ModelDiff for model reuse detection. A study on mobile deep learning apps has shown the feasibility of ModelDiff on real-world models.

Related papers

• Diffusion-TTA: Test-time Adaptation of Discriminative Models via Generative Feedback [97.0874638345205]
  generative models can be great test-time adapters for discriminative models. Our method, Diffusion-TTA, adapts pre-trained discriminative models to each unlabelled example in the test set. We show Diffusion-TTA significantly enhances the accuracy of various large-scale pre-trained discriminative models.
  arXiv  Detail & Related papers  (2023-11-27T18:59:53Z)
• Wrapper Boxes: Faithful Attribution of Model Predictions to Training Data [40.7542543934205]
  We propose a "wrapper box'' pipeline: training a neural model as usual and then using its learned feature representation in classic, interpretable models to perform prediction. Across seven language models of varying sizes, we first show that the predictive performance of wrapper classic models is largely comparable to the original neural models. Our pipeline thus preserves the predictive performance of neural language models while faithfully attributing classic model decisions to training data.
  arXiv  Detail & Related papers  (2023-11-15T01:50:53Z)
• Comparing Foundation Models using Data Kernels [13.099029073152257]
  We present a methodology for directly comparing the embedding space geometry of foundation models. Our methodology is grounded in random graph theory and enables valid hypothesis testing of embedding similarity. We show how our framework can induce a manifold of models equipped with a distance function that correlates strongly with several downstream metrics.
  arXiv  Detail & Related papers  (2023-05-09T02:01:07Z)
• Dataless Knowledge Fusion by Merging Weights of Language Models [51.8162883997512]
  Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models. This creates a barrier to fusing knowledge across individual models to yield a better single model. We propose a dataless knowledge fusion method that merges models in their parameter space.
  arXiv  Detail & Related papers  (2022-12-19T20:46:43Z)
• An Empirical Study of Deep Learning Models for Vulnerability Detection [4.243592852049963]
  We surveyed and reproduced 9 state-of-the-art deep learning models on 2 widely used vulnerability detection datasets. We investigated model capabilities, training data, and model interpretation. Our findings can help better understand model results, provide guidance on preparing training data, and improve the robustness of the models.
  arXiv  Detail & Related papers  (2022-12-15T19:49:34Z)
• ModelDiff: A Framework for Comparing Learning Algorithms [86.19580801269036]
  We study the problem of (learning) algorithm comparison, where the goal is to find differences between models trained with two different learning algorithms. We present ModelDiff, a method that leverages the datamodels framework to compare learning algorithms based on how they use their training data.
  arXiv  Detail & Related papers  (2022-11-22T18:56:52Z)
• Are You Stealing My Model? Sample Correlation for Fingerprinting Deep Neural Networks [86.55317144826179]
  Previous methods always leverage the transferable adversarial examples as the model fingerprint. We propose a novel yet simple model stealing detection method based on SAmple Correlation (SAC) SAC successfully defends against various model stealing attacks, even including adversarial training or transfer learning.
  arXiv  Detail & Related papers  (2022-10-21T02:07:50Z)
• Model Comparison in Approximate Bayesian Computation [0.456877715768796]
  A common problem in natural sciences is the comparison of competing models in the light of observed data. This framework relies on the calculation of likelihood functions which are intractable for most models used in practice. I propose a new efficient method to perform Bayesian model comparison in ABC.
  arXiv  Detail & Related papers  (2022-03-15T10:24:16Z)
• Deep Learning Models for Knowledge Tracing: Review and Empirical Evaluation [2.423547527175807]
  We review and evaluate a body of deep learning knowledge tracing (DLKT) models with openly available and widely-used data sets. The evaluated DLKT models have been reimplemented for assessing and replicability of previously reported results.
  arXiv  Detail & Related papers  (2021-12-30T14:19:27Z)
• Firearm Detection via Convolutional Neural Networks: Comparing a Semantic Segmentation Model Against End-to-End Solutions [68.8204255655161]
  Threat detection of weapons and aggressive behavior from live video can be used for rapid detection and prevention of potentially deadly incidents. One way for achieving this is through the use of artificial intelligence and, in particular, machine learning for image analysis. We compare a traditional monolithic end-to-end deep learning model and a previously proposed model based on an ensemble of simpler neural networks detecting fire-weapons via semantic segmentation.
  arXiv  Detail & Related papers  (2020-12-17T15:19:29Z)
• Data from Model: Extracting Data from Non-robust and Robust Models [83.60161052867534]
  This work explores the reverse process of generating data from a model, attempting to reveal the relationship between the data and the model. We repeat the process of Data to Model (DtM) and Data from Model (DfM) in sequence and explore the loss of feature mapping information. Our results show that the accuracy drop is limited even after multiple sequences of DtM and DfM, especially for robust models.
  arXiv  Detail & Related papers  (2020-07-13T05:27:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.