Your Space is My Zone: Demystifying the Security Risks of AI-Powered Applications on Pre-Trained Model Hubs

• URL: http://arxiv.org/abs/2606.30373v1
• Date: Mon, 29 Jun 2026 14:34:43 GMT
• Title: Your Space is My Zone: Demystifying the Security Risks of AI-Powered Applications on Pre-Trained Model Hubs
• Abstract summary: We present the first systematic security analysis of AI-Apps across three leading platforms.<n>We identify five threat categories and ten attack vectors ranging from generic web flaws to high-impact architectural issues.
• Score: 21.637785874073444
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI-powered Applications (AI-Apps), hosted on platforms such as Hugging Face, are democratizing access to pre-trained models through online inference and fine-tuning services. While lowering AI adoption barriers, these platforms introduce an unexplored attack surface, as AI-Apps are often developed by untrusted parties with weak isolation and misconfigured security settings. In this paper, we present the first systematic security analysis of AI-Apps across three leading platforms. To structure our investigation, we map the AI-App lifecycle to established risk taxonomies (e.g., OWASP), identifying five threat categories and ten attack vectors ranging from generic web flaws to high-impact architectural issues. Our analysis reveals critical failures including broken access control, insecure resource reuse, insufficient input validation, and sensitive data exposure. Notably, we uncover three novel architectural vulnerabilities inherent to platform design and demonstrate how traditional issues (e.g., world-readable logs) are uniquely amplified in this ecosystem. To assess real-world impact, we develop an analysis framework Insightor and apply it to over 970,000 public AI-Apps. Alarmingly, we find thousands of apps leaking credentials, hundreds containing input injection vulnerabilities that allow arbitrary code execution, and tens harboring embedded backdoors -- indicating active exploitation. We have responsibly disclosed all findings to the affected platforms and developers.

Related papers

• A Measurement Study of Cryptographic Misuse in Embodied AI Mobile Applications [18.86199323484397]
  Embodied AI (EAI) mobile applications are evolving from auxiliary user interfaces into active control-path components.<n>We present the first large-scale measurement study of cryptographic misuse within the EAI mobile ecosystem.
  arXiv  Detail & Related papers  (2026-06-18T09:24:22Z)
• ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? [92.21756459993695]
  Exploitation is a challenging task because it requires low-level program reasoning.<n>Despite its importance and diagnostic value, exploitation remains under-evaluated.<n>We introduce ExploitGym, a large-scale, diverse, realistic benchmark on the exploitation capabilities of AI agents.
  arXiv  Detail & Related papers  (2026-05-11T18:00:14Z)
• SoK: The Attack Surface of Agentic AI -- Tools, and Autonomy [3.340255811686752]
  Recent AI systems combine large language models with tools, external knowledge via retrieval-augmented generation (RAG)<n>This agentic AI paradigm greatly expands capabilities - but also vastly enlarges the attack surface.<n>We develop a comprehensive taxonomy of attacks spanning prompt-level injections, knowledge-base poisoning, tool/plug-in exploits, and multi-agent emergent threats.
  arXiv  Detail & Related papers  (2026-03-24T08:21:51Z)
• Cisco Integrated AI Security and Safety Framework Report [3.162988913169078]
  This paper presents Cisco's Integrated AI Security and Safety Framework ("AI Security Framework")<n>The framework can be used to classify, integrate, and operationalize the full range of AI risks.<n>It integrates AI security and AI safety across modalities, agents, pipelines, and the broader ecosystem.
  arXiv  Detail & Related papers  (2025-12-15T02:12:12Z)
• An Empirical Study on the Security Vulnerabilities of GPTs [48.12756684275687]
  GPTs are one kind of customized AI agents based on OpenAI's large language models.<n>We present an empirical study on the security vulnerabilities of GPTs.
  arXiv  Detail & Related papers  (2025-11-28T13:30:25Z)
• BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agents [8.923854146974783]
  We examine the landscape of prompt injection attacks and synthesize a benchmark of attacks embedded in realistic HTML payloads.<n>Our benchmark goes beyond prior work by emphasizing injections that can influence real-world actions rather than mere text outputs.<n>We propose a multi-layered defense strategy comprising both architectural and model-based defenses.
  arXiv  Detail & Related papers  (2025-11-25T18:28:35Z)
• ANNIE: Be Careful of Your Robots [48.89876809734855]
  We present the first systematic study of adversarial safety attacks on embodied AI systems.<n>We show attack success rates exceeding 50% across all safety categories.<n>Results expose a previously underexplored but highly consequential attack surface in embodied AI systems.
  arXiv  Detail & Related papers  (2025-09-03T15:00:28Z)
• A Systematic Survey of Model Extraction Attacks and Defenses: State-of-the-Art and Perspectives [65.3369988566853]
  Recent studies have demonstrated that adversaries can replicate a target model's functionality.<n>Model Extraction Attacks pose threats to intellectual property, privacy, and system security.<n>We propose a novel taxonomy that classifies MEAs according to attack mechanisms, defense approaches, and computing environments.
  arXiv  Detail & Related papers  (2025-08-20T19:49:59Z)
• ASTRA: Autonomous Spatial-Temporal Red-teaming for AI Software Assistants [21.35387344588118]
  ASTRA is an automated system designed to uncover safety flaws in AI-driven code generation and security guidance systems.<n>ASTRA finds 11-66% more issues than existing techniques and produces test cases that lead to 17% more effective alignment training.
  arXiv  Detail & Related papers  (2025-08-05T21:57:52Z)
• Real AI Agents with Fake Memories: Fatal Context Manipulation Attacks on Web3 Agents [36.49717045080722]
  This paper investigates the vulnerabilities of AI agents within blockchain-based financial ecosystems when exposed to adversarial threats in real-world scenarios.<n>We introduce the concept of context manipulation -- a comprehensive attack vector that exploits unprotected context surfaces.<n>Using ElizaOS, we showcase that malicious injections into prompts or historical records can trigger unauthorized asset transfers and protocol violations.
  arXiv  Detail & Related papers  (2025-03-20T15:44:31Z)
• When Authentication Is Not Enough: On the Security of Behavioral-Based Driver Authentication Systems [53.2306792009435]
  We develop two lightweight driver authentication systems based on Random Forest and Recurrent Neural Network architectures. We are the first to propose attacks against these systems by developing two novel evasion attacks, SMARTCAN and GANCAN. Through our contributions, we aid practitioners in safely adopting these systems, help reduce car thefts, and enhance driver security.
  arXiv  Detail & Related papers  (2023-06-09T14:33:26Z)
• An Empirical Study of AI Techniques in Mobile Applications [10.43634556488264]
  We conducted the most extensive empirical study on AI applications, exploring on-device ML apps, on-device DL apps, and AI service-supported (cloud-based) apps. Our study has strong implications for AI app developers, users, and AI R&D.
  arXiv  Detail & Related papers  (2022-12-03T15:31:34Z)
• Proceedings of the Artificial Intelligence for Cyber Security (AICS) Workshop at AAAI 2022 [55.573187938617636]
  The workshop will focus on the application of AI to problems in cyber security. Cyber systems generate large volumes of data, utilizing this effectively is beyond human capabilities.
  arXiv  Detail & Related papers  (2022-02-28T18:27:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.