Models Are Codes: Towards Measuring Malicious Code Poisoning Attacks on Pre-trained Model Hubs

• URL: http://arxiv.org/abs/2409.09368v1
• Date: Sat, 14 Sep 2024 08:47:22 GMT
• Title: Models Are Codes: Towards Measuring Malicious Code Poisoning Attacks on Pre-trained Model Hubs
• Authors: Jian Zhao, Shenao Wang, Yanjie Zhao, Xinyi Hou, Kailong Wang, Peiming Gao, Yuanchao Zhang, Chen Wei, Haoyu Wang,
• Abstract summary: This paper presents the first systematic study of malicious code poisoning attacks on pre-trained model hubs, focusing on the Hugging Face platform. We propose MalHug, an end-to-end pipeline tailored for Hugging Face that combines dataset loading script extraction, model deserialization, and taint pattern matching. MalHug has monitored more than 705K models and 176K datasets, uncovering 91 malicious models and 9 malicious dataset loading scripts.
• Score: 10.252989233081395
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The proliferation of pre-trained models (PTMs) and datasets has led to the emergence of centralized model hubs like Hugging Face, which facilitate collaborative development and reuse. However, recent security reports have uncovered vulnerabilities and instances of malicious attacks within these platforms, highlighting growing security concerns. This paper presents the first systematic study of malicious code poisoning attacks on pre-trained model hubs, focusing on the Hugging Face platform. We conduct a comprehensive threat analysis, develop a taxonomy of model formats, and perform root cause analysis of vulnerable formats. While existing tools like Fickling and ModelScan offer some protection, they face limitations in semantic-level analysis and comprehensive threat detection. To address these challenges, we propose MalHug, an end-to-end pipeline tailored for Hugging Face that combines dataset loading script extraction, model deserialization, in-depth taint analysis, and heuristic pattern matching to detect and classify malicious code poisoning attacks in datasets and models. In collaboration with Ant Group, a leading financial technology company, we have implemented and deployed MalHug on a mirrored Hugging Face instance within their infrastructure, where it has been operational for over three months. During this period, MalHug has monitored more than 705K models and 176K datasets, uncovering 91 malicious models and 9 malicious dataset loading scripts. These findings reveal a range of security threats, including reverse shell, browser credential theft, and system reconnaissance. This work not only bridges a critical gap in understanding the security of the PTM supply chain but also provides a practical, industry-tested solution for enhancing the security of pre-trained model hubs.

Related papers

• Zero-Trust Artificial Intelligence Model Security Based on Moving Target Defense and Content Disarm and Reconstruction [4.0208298639821525]
  This paper examines the challenges in distributing AI models through model zoos and file transfer mechanisms. The physical security of model files is critical, requiring stringent access controls and attack prevention solutions. It demonstrates a 100% disarm rate while validated against known AI model repositories and actual malware attacks from the HuggingFace model zoo.
  arXiv  Detail & Related papers  (2025-03-03T17:32:19Z)
• Merger-as-a-Stealer: Stealing Targeted PII from Aligned LLMs with Model Merging [49.270050440553575]
  We propose textttMerger-as-a-Stealer, a two-stage framework to achieve this attack. First, the attacker fine-tunes a malicious model to force it to respond to any PII-related queries. Second, the attacker inputs direct PII-related queries to the merged model to extract targeted PII.
  arXiv  Detail & Related papers  (2025-02-22T05:34:53Z)
• Model-agnostic clean-label backdoor mitigation in cybersecurity environments [6.857489153636145]
  Recent research has surfaced a series of insidious training-time attacks that inject backdoors in models designed for security classification tasks. We propose new techniques that leverage insights in cybersecurity threat models to effectively mitigate these clean-label poisoning attacks.
  arXiv  Detail & Related papers  (2024-07-11T03:25:40Z)
• FullCert: Deterministic End-to-End Certification for Training and Inference of Neural Networks [62.897993591443594]
  FullCert is the first end-to-end certifier with sound, deterministic bounds. We experimentally demonstrate FullCert's feasibility on two datasets.
  arXiv  Detail & Related papers  (2024-06-17T13:23:52Z)
• Mellivora Capensis: A Backdoor-Free Training Framework on the Poisoned Dataset without Auxiliary Data [29.842087372804905]
  This paper addresses the challenges of backdoor attack countermeasures in real-world scenarios. We propose a robust and clean-data-free backdoor defense framework, namely Mellivora Capensis (textttMeCa), which enables the model trainer to train a clean model on the poisoned dataset.
  arXiv  Detail & Related papers  (2024-05-21T12:20:19Z)
• Measuring Impacts of Poisoning on Model Parameters and Embeddings for Large Language Models of Code [4.305373051747465]
  Large language models (LLMs) have revolutionized software development practices, yet concerns about their safety have arisen. Backdoor attacks involve the insertion of triggers into training data, allowing attackers to manipulate the behavior of the model maliciously. In this paper, we focus on analyzing the model parameters to detect potential backdoor signals in code models.
  arXiv  Detail & Related papers  (2024-05-19T06:53:20Z)
• Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models [112.48136829374741]
  In this paper, we unveil a new vulnerability: the privacy backdoor attack. When a victim fine-tunes a backdoored model, their training data will be leaked at a significantly higher rate than if they had fine-tuned a typical model. Our findings highlight a critical privacy concern within the machine learning community and call for a reevaluation of safety protocols in the use of open-source pre-trained models.
  arXiv  Detail & Related papers  (2024-04-01T16:50:54Z)
• Unlearning Backdoor Threats: Enhancing Backdoor Defense in Multimodal Contrastive Learning via Local Token Unlearning [49.242828934501986]
  Multimodal contrastive learning has emerged as a powerful paradigm for building high-quality features. backdoor attacks subtly embed malicious behaviors within the model during training. We introduce an innovative token-based localized forgetting training regime.
  arXiv  Detail & Related papers  (2024-03-24T18:33:15Z)
• Model X-ray:Detecting Backdoored Models via Decision Boundary [62.675297418960355]
  Backdoor attacks pose a significant security vulnerability for deep neural networks (DNNs) We propose Model X-ray, a novel backdoor detection approach based on the analysis of illustrated two-dimensional (2D) decision boundaries. Our approach includes two strategies focused on the decision areas dominated by clean samples and the concentration of label distribution.
  arXiv  Detail & Related papers  (2024-02-27T12:42:07Z)
• CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Models [58.27254444280376]
  Large language models (LLMs) for automatic code generation have achieved breakthroughs in several programming tasks. Training data for these models is usually collected from the Internet (e.g., from open-source repositories) and is likely to contain faults and security vulnerabilities. This unsanitized training data can cause the language models to learn these vulnerabilities and propagate them during the code generation procedure.
  arXiv  Detail & Related papers  (2023-02-08T11:54:07Z)
• A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks [72.7373468905418]
  We develop an open-source toolkit OpenBackdoor to foster the implementations and evaluations of textual backdoor learning. We also propose CUBE, a simple yet strong clustering-based defense baseline.
  arXiv  Detail & Related papers  (2022-06-17T02:29:23Z)
• Poisoning Attacks and Defenses on Artificial Intelligence: A Survey [3.706481388415728]
  Data poisoning attacks represent a type of attack that consists of tampering the data samples fed to the model during the training phase, leading to a degradation in the models accuracy during the inference phase. This work compiles the most relevant insights and findings found in the latest existing literatures addressing this type of attacks. A thorough assessment is performed on the reviewed works, comparing the effects of data poisoning on a wide range of ML models in real-world conditions.
  arXiv  Detail & Related papers  (2022-02-21T14:43:38Z)
• Model Extraction Attacks on Graph Neural Networks: Taxonomy and Realization [40.37373934201329]
  We investigate and develop model extraction attacks against GNN models. We first formalise the threat modelling in the context of GNN model extraction. We then present detailed methods which utilise the accessible knowledge in each threat to implement the attacks.
  arXiv  Detail & Related papers  (2020-10-24T03:09:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.