Deep Dive into the Abuse of DL APIs To Create Malicious AI Models and How to Detect Them

• URL: http://arxiv.org/abs/2601.04553v1
• Date: Thu, 08 Jan 2026 03:30:20 GMT
• Title: Deep Dive into the Abuse of DL APIs To Create Malicious AI Models and How to Detect Them
• Authors: Mohamed Nabeel, Oleksii Starov,
• Abstract summary: Pre-trained AI models are often fetched from model hubs like Hugging Face or Hub.<n>This introduces a security risk where attackers can inject malicious code into the models.<n>We show how one can abuse hidden functionalities of APIs such as file read/write and network send/receive.
• Score: 3.8802542855314788
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: According to Gartner, more than 70% of organizations will have integrated AI models into their workflows by the end of 2025. In order to reduce cost and foster innovation, it is often the case that pre-trained models are fetched from model hubs like Hugging Face or TensorFlow Hub. However, this introduces a security risk where attackers can inject malicious code into the models they upload to these hubs, leading to various kinds of attacks including remote code execution (RCE), sensitive data exfiltration, and system file modification when these models are loaded or executed (predict function). Since AI models play a critical role in digital transformation, this would drastically increase the number of software supply chain attacks. While there are several efforts at detecting malware when deserializing pickle based saved models (hiding malware in model parameters), the risk of abusing DL APIs (e.g. TensorFlow APIs) is understudied. Specifically, we show how one can abuse hidden functionalities of TensorFlow APIs such as file read/write and network send/receive along with their persistence APIs to launch attacks. It is concerning to note that existing scanners in model hubs like Hugging Face and TensorFlow Hub are unable to detect some of the stealthy abuse of such APIs. This is because scanning tools only have a syntactically identified set of suspicious functionality that is being analysed. They often do not have a semantic-level understanding of the functionality utilized. After demonstrating the possible attacks, we show how one may identify potentially abusable hidden API functionalities using LLMs and build scanners to detect such abuses.

Related papers

• Self-Purification Mitigates Backdoors in Multimodal Diffusion Language Models [74.1970982768771]
  We show that well-established data-poisoning pipelines can successfully implant backdoors into MDLMs.<n>We introduce a backdoor defense framework for MDLMs named DiSP (Diffusion Self-Purification)
  arXiv  Detail & Related papers  (2026-02-24T15:47:52Z)
• CREDIT: Certified Ownership Verification of Deep Neural Networks Against Model Extraction Attacks [54.04030169323115]
  We introduce CREDIT, a certified ownership verification against Model Extraction Attacks (MEAs)<n>We quantify the similarity between DNN models, propose a practical verification threshold, and provide rigorous theoretical guarantees for ownership verification based on this threshold.<n>We extensively evaluate our approach on several mainstream datasets across different domains and tasks, achieving state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-02-23T23:36:25Z)
• Assimilation Matters: Model-level Backdoor Detection in Vision-Language Pretrained Models [71.44858461725893]
  Given a model fine-tuned by an untrusted third party, determining whether the model has been injected with a backdoor is a critical and challenging problem.<n>Existing detection methods usually rely on prior knowledge of training dataset, backdoor triggers and targets.<n>We introduce Assimilation Matters in DETection (AMDET), a novel model-level detection framework that operates without any such prior knowledge.
  arXiv  Detail & Related papers  (2025-11-29T06:20:00Z)
• Fundamental Limitations in Pointwise Defences of LLM Finetuning APIs [56.84338097581665]
  We show that defences of fine-tuning APIs are fundamentally limited in their ability to prevent fine-tuning attacks.<n>We construct 'pointwise-undetectable' attacks that repurpose entropy in benign model outputs to covertly transmit dangerous knowledge.<n>We test our attacks against the OpenAI fine-tuning API, finding they succeed in eliciting answers to harmful multiple-choice questions.
  arXiv  Detail & Related papers  (2025-02-20T18:45:01Z)
• Malware Detection based on API calls [0.48866322421122627]
  We explore a lightweight, order-invariant approach to detecting and mitigating malware threats.<n>We publish a public dataset of over three hundred thousand samples, annotated with labels indicating benign or malicious activity.<n>We leverage machine learning algorithms, such as random forests, and conduct behavioral analysis by examining patterns and anomalies in API call sequences.
  arXiv  Detail & Related papers  (2025-02-18T13:51:56Z)
• Bounding-box Watermarking: Defense against Model Extraction Attacks on Object Detectors [1.1510009152620668]
  This work focuses on object detection (OD) models.<n>Existing backdoor attacks on OD models are not applicable for model watermarking as the defense against MEAs on a realistic threat model.<n>Our proposed approach involves inserting a backdoor into extracted models via APIs by stealthily modifying the bounding-boxes (BBs) of objects detected in queries while keeping the OD capability.
  arXiv  Detail & Related papers  (2024-11-20T05:40:20Z)
• A Classification-by-Retrieval Framework for Few-Shot Anomaly Detection to Detect API Injection Attacks [9.693391036125908]
  We propose a novel unsupervised few-shot anomaly detection framework composed of two main parts. First, we train a dedicated generic language model for API based on FastText embedding. Next, we use Approximate Nearest Neighbor search in a classification-by-retrieval approach.
  arXiv  Detail & Related papers  (2024-05-18T10:15:31Z)
• Model X-ray:Detecting Backdoored Models via Decision Boundary [62.675297418960355]
  Backdoor attacks pose a significant security vulnerability for deep neural networks (DNNs) We propose Model X-ray, a novel backdoor detection approach based on the analysis of illustrated two-dimensional (2D) decision boundaries. Our approach includes two strategies focused on the decision areas dominated by clean samples and the concentration of label distribution.
  arXiv  Detail & Related papers  (2024-02-27T12:42:07Z)
• Prompt Engineering-assisted Malware Dynamic Analysis Using GPT-4 [45.935748395725206]
  We introduce a prompt engineering-assisted malware dynamic analysis using GPT-4. In this method, GPT-4 is employed to create explanatory text for each API call within the API sequence. BERT is used to obtain the representation of the text, from which we derive the representation of the API sequence.
  arXiv  Detail & Related papers  (2023-12-13T17:39:44Z)
• Mask and Restore: Blind Backdoor Defense at Test Time with Masked Autoencoder [50.1394620328318]
  Existing backdoor defense methods often require accessing a few validation data and model parameters.<n>We propose blind backdoor defense with Masked AutoEncoder (BDMAE)<n>BDMAE detects possible local triggers using image structural similarity and label consistency between the test image and MAE restorations.
  arXiv  Detail & Related papers  (2023-03-27T19:23:33Z)
• CrowdGuard: Federated Backdoor Detection in Federated Learning [39.58317527488534]
  This paper presents a novel defense mechanism, CrowdGuard, that effectively mitigates backdoor attacks in Federated Learning. CrowdGuard employs a server-located stacked clustering scheme to enhance its resilience to rogue client feedback. The evaluation results demonstrate that CrowdGuard achieves a 100% True-Positive-Rate and True-Negative-Rate across various scenarios.
  arXiv  Detail & Related papers  (2022-10-14T11:27:49Z)
• DeepSight: Mitigating Backdoor Attacks in Federated Learning Through Deep Model Inspection [26.593268413299228]
  Federated Learning (FL) allows multiple clients to collaboratively train a Neural Network (NN) model on their private data without revealing the data. DeepSight is a novel model filtering approach for mitigating backdoor attacks. We show that it can mitigate state-of-the-art backdoor attacks with a negligible impact on the model's performance on benign data.
  arXiv  Detail & Related papers  (2022-01-03T17:10:07Z)
• Defending against Model Stealing via Verifying Embedded External Features [90.29429679125508]
  adversaries can steal' deployed models even when they have no training samples and can not get access to the model parameters or structures. We explore the defense from another angle by verifying whether a suspicious model contains the knowledge of defender-specified emphexternal features. Our method is effective in detecting different types of model stealing simultaneously, even if the stolen model is obtained via a multi-stage stealing process.
  arXiv  Detail & Related papers  (2021-12-07T03:51:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.