Breaking SafetyCore: Exploring the Risks of On-Device AI Deployment

• URL: http://arxiv.org/abs/2509.06371v1
• Date: Mon, 08 Sep 2025 06:53:13 GMT
• Title: Breaking SafetyCore: Exploring the Risks of On-Device AI Deployment
• Authors: Victor Guyomard, Mathis Mauvisseau, Marie Paindavoine,
• Abstract summary: An increasing number of AI models are deployed on-device.<n>This shift enhances privacy and reduces latency, but also introduces security risks distinct from traditional software.<n>In this article, we examine these risks through the real-world case study of SafetyCore, an Android system service incorporating sensitive image content detection.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Due to hardware and software improvements, an increasing number of AI models are deployed on-device. This shift enhances privacy and reduces latency, but also introduces security risks distinct from traditional software. In this article, we examine these risks through the real-world case study of SafetyCore, an Android system service incorporating sensitive image content detection. We demonstrate how the on-device AI model can be extracted and manipulated to bypass detection, effectively rendering the protection ineffective. Our analysis exposes vulnerabilities of on-device AI models and provides a practical demonstration of how adversaries can exploit them.

Related papers

• Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report v1.5 [61.787178868669265]
  This technical report presents an updated and granular assessment of five critical dimensions: cyber offense, persuasion and manipulation, strategic deception, uncontrolled AI R&D, and self-replication.<n>This work reflects our current understanding of AI frontier risks and urges collective action to mitigate these challenges.
  arXiv  Detail & Related papers  (2026-02-16T04:30:06Z)
• OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflows [77.95511352806261]
  Computer-using agents powered by Vision-Language Models (VLMs) have demonstrated human-like capabilities in operating digital environments like mobile platforms.<n>We propose OS-Sentinel, a novel hybrid safety detection framework that combines a Formal Verifier for detecting explicit system-level violations with a Contextual Judge for assessing contextual risks and agent actions.
  arXiv  Detail & Related papers  (2025-10-28T13:22:39Z)
• Qualcomm Trusted Application Emulation for Fuzzing Testing [0.3277163122167433]
  This research centers on trusted applications (TAs) within the Qualcomm TEE.<n>Through reverse engineering techniques, we develop a partial emulation environment that accurately emulates their behavior.<n>We integrate fuzzing testing techniques into the emulator to systematically uncover potential vulnerabilities within Qualcomm TAs.
  arXiv  Detail & Related papers  (2025-07-11T06:10:15Z)
• Frontier AI's Impact on the Cybersecurity Landscape [46.32458228179959]
  We find that while AI is already widely used in attacks, its application in defense remains limited.<n>Experts expect AI to continue favoring attackers over defenders, though the gap will gradually narrow.
  arXiv  Detail & Related papers  (2025-04-07T18:25:18Z)
• An Approach to Technical AGI Safety and Security [72.83728459135101]
  We develop an approach to address the risk of harms consequential enough to significantly harm humanity.<n>We focus on technical approaches to misuse and misalignment.<n>We briefly outline how these ingredients could be combined to produce safety cases for AGI systems.
  arXiv  Detail & Related papers  (2025-04-02T15:59:31Z)
• Runtime Detection of Adversarial Attacks in AI Accelerators Using Performance Counters [5.097354139604596]
  We propose SAMURAI, a novel framework for safeguarding against malicious usage of AI hardware.<n> SAMURAI introduces an AI Performance Counter ( APC) for tracking dynamic behavior of an AI model.<n> APC records the runtime profile of the low-level hardware events of different AI operations.<n>The summary information recorded by the APC is processed by TANTO to efficiently identify potential security breaches.
  arXiv  Detail & Related papers  (2025-03-10T17:38:42Z)
• Computational Safety for Generative AI: A Signal Processing Perspective [65.268245109828]
  computational safety is a mathematical framework that enables the quantitative assessment, formulation, and study of safety challenges in GenAI.<n>We show how sensitivity analysis and loss landscape analysis can be used to detect malicious prompts with jailbreak attempts.<n>We discuss key open research challenges, opportunities, and the essential role of signal processing in computational AI safety.
  arXiv  Detail & Related papers  (2025-02-18T02:26:50Z)
• OpenAI o1 System Card [274.83891368890977]
  The o1 model series is trained with large-scale reinforcement learning to reason using chain of thought.<n>This report outlines the safety work carried out for the OpenAI o1 and OpenAI o1-mini models, including safety evaluations, external red teaming, and Preparedness Framework evaluations.
  arXiv  Detail & Related papers  (2024-12-21T18:04:31Z)
• Position: Mind the Gap-the Growing Disconnect Between Established Vulnerability Disclosure and AI Security [56.219994752894294]
  We argue that adapting existing processes for AI security reporting is doomed to fail due to fundamental shortcomings for the distinctive characteristics of AI systems.<n>Based on our proposal to address these shortcomings, we discuss an approach to AI security reporting and how the new AI paradigm, AI agents, will further reinforce the need for specialized AI security incident reporting advancements.
  arXiv  Detail & Related papers  (2024-12-19T13:50:26Z)
• Modeling Electromagnetic Signal Injection Attacks on Camera-based Smart Systems: Applications and Mitigation [18.909937495767313]
  electromagnetic waves pose a threat to safety- or security-critical systems. Such attacks enable attackers to manipulate the images remotely, leading to incorrect AI decisions. We present a pilot study on adversarial training to improve their robustness against attacks.
  arXiv  Detail & Related papers  (2024-08-09T15:33:28Z)
• Artificial Intelligence as the New Hacker: Developing Agents for Offensive Security [0.0]
  This paper explores the integration of Artificial Intelligence (AI) into offensive cybersecurity. It develops an autonomous AI agent, ReaperAI, designed to simulate and execute cyberattacks. ReaperAI demonstrates the potential to identify, exploit, and analyze security vulnerabilities autonomously.
  arXiv  Detail & Related papers  (2024-05-09T18:15:12Z)
• A Unified Hardware-based Threat Detector for AI Accelerators [12.96840649714218]
  We design UniGuard, a novel unified and non-intrusive detection methodology to safeguard FPGA-based AI accelerators. We employ a Time-to-Digital Converter to capture power fluctuations and train a supervised machine learning model to identify various types of threats.
  arXiv  Detail & Related papers  (2023-11-28T10:55:02Z)
• When Authentication Is Not Enough: On the Security of Behavioral-Based Driver Authentication Systems [53.2306792009435]
  We develop two lightweight driver authentication systems based on Random Forest and Recurrent Neural Network architectures. We are the first to propose attacks against these systems by developing two novel evasion attacks, SMARTCAN and GANCAN. Through our contributions, we aid practitioners in safely adopting these systems, help reduce car thefts, and enhance driver security.
  arXiv  Detail & Related papers  (2023-06-09T14:33:26Z)
• Dos and Don'ts of Machine Learning in Computer Security [74.1816306998445]
  Despite great potential, machine learning in security is prone to subtle pitfalls that undermine its performance. We identify common pitfalls in the design, implementation, and evaluation of learning-based security systems. We propose actionable recommendations to support researchers in avoiding or mitigating the pitfalls where possible.
  arXiv  Detail & Related papers  (2020-10-19T13:09:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.