Don't Trust the AI Ecosystem: Analyzing Privacy Leakage in Compromised Open-Source Components

• URL: http://arxiv.org/abs/2607.27886v2
• Date: Fri, 31 Jul 2026 01:31:02 GMT
• Title: Don't Trust the AI Ecosystem: Analyzing Privacy Leakage in Compromised Open-Source Components
• Abstract summary: We introduce GradLock, a training-time injection attack that stealthily injects sensitive training data directly into the model parameters.<n>Experiments on MNIST, Imagenette, and CelebA demonstrate that GradLock achieves near-lossless reconstruction (SSIM 1.0) and instant extraction ( 1.0s)
• Score: 1.6166549398149002
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing model inversion (MI) attacks predominantly rely on post-training optimization to recover private data from model outputs. However, these methods are fundamentally constrained by the target model's generalization bottleneck, often yielding generic features rather than specific identities, particularly on high-dimensional datasets. In this paper, we introduce GradLock, a novel training-time injection attack that stealthily injects sensitive training data directly into the model parameters. Operating within a compromised supply chain context, GradLock leverages stateless deterministic indexing to establish isolated data vaults and employs dynamic gradient locking to prevent payload degradation during the optimization process. This mechanism allows the adversary to extract pixel-perfect data from the final model without retaining access to the training environment. Extensive experiments on MNIST, Imagenette, and CelebA demonstrate that GradLock achieves near-lossless reconstruction (SSIM ~ 1.0) and instant extraction (< 1.0s). Compared to existing training-time injection methods, our approach exhibits superior robustness against standard deployment optimizations, including quantization, pruning, and fine-tuning. Furthermore, a user deployment study reveals that 93.3% of participants failed to detect the malicious logic, highlighting a severe blind spot in the security of modern AI supply chains.

Related papers

• FL-PBM: Pre-Training Backdoor Mitigation for Federated Learning [15.911410230033816]
  Backdoor attacks pose a significant threat to the integrity and reliability of Artificial Intelligence (AI) models.<n>In this paper, we propose Pre-Training Backdoor Mitigation for Federated Learning.<n>It proactively filters poisoned data on the client side before model training in a federated learning environment.
  arXiv  Detail & Related papers  (2026-03-30T16:56:38Z)
• Deep Leakage with Generative Flow Matching Denoiser [54.05993847488204]
  We introduce a new deep leakage (DL) attack that integrates a generative Flow Matching (FM) prior into the reconstruction process.<n>Our approach consistently outperforms state-of-the-art attacks across pixel-level, perceptual, and feature-based similarity metrics.
  arXiv  Detail & Related papers  (2026-01-21T14:51:01Z)
• GShield: Mitigating Poisoning Attacks in Federated Learning [2.6260952524631787]
  Federated Learning (FL) has recently emerged as a revolutionary approach to collaborative training Machine Learning models.<n>It enables decentralized model training while preserving data privacy, but its distributed nature makes it highly vulnerable to a severe attack known as Data Poisoning.<n>We present a novel defense mechanism called GShield, designed to detect and mitigate malicious and low-quality updates.
  arXiv  Detail & Related papers  (2025-12-22T11:29:28Z)
• Non-Linear Trajectory Modeling for Multi-Step Gradient Inversion Attacks in Federated Learning [16.19043018432204]
  We propose Non-Linear Surrogate Model Extension (NL-SME), the first method to introduce nonlinear parametric trajectory modeling for Gradient Inversion Attacks (GIAs)<n>Our approach replaces linear pose with learnable quadratic B'ezier curves that capture SGD's curved characteristics through control points, combined with regularization and dvec scaling mechanisms for enhanced expressiveness.
  arXiv  Detail & Related papers  (2025-09-26T09:04:25Z)
• Adversarial Robustification via Text-to-Image Diffusion Models [56.37291240867549]
  Adrial robustness has been conventionally believed as a challenging property to encode for neural networks. We develop a scalable and model-agnostic solution to achieve adversarial robustness without using any data.
  arXiv  Detail & Related papers  (2024-07-26T10:49:14Z)
• Nonlinear Transformations Against Unlearnable Datasets [4.876873339297269]
  Automated scraping stands out as a common method for collecting data in deep learning models without the authorization of data owners. Recent studies have begun to tackle the privacy concerns associated with this data collection method. The data generated by those approaches, called "unlearnable" examples, are prevented "learning" by deep learning models.
  arXiv  Detail & Related papers  (2024-06-05T03:00:47Z)
• Enhancing Multiple Reliability Measures via Nuisance-extended Information Bottleneck [77.37409441129995]
  In practical scenarios where training data is limited, many predictive signals in the data can be rather from some biases in data acquisition. We consider an adversarial threat model under a mutual information constraint to cover a wider class of perturbations in training. We propose an autoencoder-based training to implement the objective, as well as practical encoder designs to facilitate the proposed hybrid discriminative-generative training.
  arXiv  Detail & Related papers  (2023-03-24T16:03:21Z)
• MAPS: A Noise-Robust Progressive Learning Approach for Source-Free Domain Adaptive Keypoint Detection [76.97324120775475]
  Cross-domain keypoint detection methods always require accessing the source data during adaptation. This paper considers source-free domain adaptive keypoint detection, where only the well-trained source model is provided to the target domain.
  arXiv  Detail & Related papers  (2023-02-09T12:06:08Z)
• Do Gradient Inversion Attacks Make Federated Learning Unsafe? [70.0231254112197]
  Federated learning (FL) allows the collaborative training of AI models without needing to share raw data. Recent works on the inversion of deep neural networks from model gradients raised concerns about the security of FL in preventing the leakage of training data. In this work, we show that these attacks presented in the literature are impractical in real FL use-cases and provide a new baseline attack.
  arXiv  Detail & Related papers  (2022-02-14T18:33:12Z)
• Contrastive Model Inversion for Data-Free Knowledge Distillation [60.08025054715192]
  We propose Contrastive Model Inversion, where the data diversity is explicitly modeled as an optimizable objective. Our main observation is that, under the constraint of the same amount of data, higher data diversity usually indicates stronger instance discrimination. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that CMI achieves significantly superior performance when the generated data are used for knowledge distillation.
  arXiv  Detail & Related papers  (2021-05-18T15:13:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.