Learned Systems Security

• URL: http://arxiv.org/abs/2212.10318v2
• Date: Thu, 22 Dec 2022 17:36:03 GMT
• Title: Learned Systems Security
• Authors: Roei Schuster, Jin Peng Zhou, Thorsten Eisenhofer, Paul Grubbs, Nicolas Papernot
• Abstract summary: A learned system uses machine learning (ML) internally to improve performance. We can expect such systems to be vulnerable to some adversarial-ML attacks. We develop a framework for identifying vulnerabilities that stem from the use of ML.
• Score: 30.39158287782567
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A learned system uses machine learning (ML) internally to improve performance. We can expect such systems to be vulnerable to some adversarial-ML attacks. Often, the learned component is shared between mutually-distrusting users or processes, much like microarchitectural resources such as caches, potentially giving rise to highly-realistic attacker models. However, compared to attacks on other ML-based systems, attackers face a level of indirection as they cannot interact directly with the learned model. Additionally, the difference between the attack surface of learned and non-learned versions of the same system is often subtle. These factors obfuscate the de-facto risks that the incorporation of ML carries. We analyze the root causes of potentially-increased attack surface in learned systems and develop a framework for identifying vulnerabilities that stem from the use of ML. We apply our framework to a broad set of learned systems under active development. To empirically validate the many vulnerabilities surfaced by our framework, we choose 3 of them and implement and evaluate exploits against prominent learned-system instances. We show that the use of ML caused leakage of past queries in a database, enabled a poisoning attack that causes exponential memory blowup in an index structure and crashes it in seconds, and enabled index users to snoop on each others' key distributions by timing queries over their own keys. We find that adversarial ML is a universal threat against learned systems, point to open research gaps in our understanding of learned-systems security, and conclude by discussing mitigations, while noting that data leakage is inherent in systems whose learned component is shared between multiple parties.

Related papers

• "Glue pizza and eat rocks" -- Exploiting Vulnerabilities in Retrieval-Augmented Generative Models [74.05368440735468]
  Retrieval-Augmented Generative (RAG) models enhance Large Language Models (LLMs) In this paper, we demonstrate a security threat where adversaries can exploit the openness of these knowledge bases.
  arXiv  Detail & Related papers  (2024-06-26T05:36:23Z)
• On Security Weaknesses and Vulnerabilities in Deep Learning Systems [32.14068820256729]
  We specifically look into deep learning (DL) framework and perform the first systematic study of vulnerabilities in DL systems. We propose a two-stream data analysis framework to explore vulnerability patterns from various databases. We conducted a large-scale empirical study of 3,049 DL vulnerabilities to better understand the patterns of vulnerability and the challenges in fixing them.
  arXiv  Detail & Related papers  (2024-06-12T23:04:13Z)
• Transfer Learning in Pre-Trained Large Language Models for Malware Detection Based on System Calls [3.5698678013121334]
  This work presents a novel framework leveraging large language models (LLMs) to classify malware based on system call data. Experiments with a dataset of over 1TB of system calls demonstrate that models with larger context sizes, such as BigBird and Longformer, achieve superior accuracy and F1-Score of approximately 0.86. This approach shows significant potential for real-time detection in high-stakes environments, offering a robust solution to evolving cyber threats.
  arXiv  Detail & Related papers  (2024-05-15T13:19:43Z)
• Vulnerability of Machine Learning Approaches Applied in IoT-based Smart Grid: A Review [51.31851488650698]
  Machine learning (ML) sees an increasing prevalence of being used in the internet-of-things (IoT)-based smart grid. adversarial distortion injected into the power signal will greatly affect the system's normal control and operation. It is imperative to conduct vulnerability assessment for MLsgAPPs applied in the context of safety-critical power systems.
  arXiv  Detail & Related papers  (2023-08-30T03:29:26Z)
• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection [64.67495502772866]
  Large Language Models (LLMs) are increasingly being integrated into various applications. We show how attackers can override original instructions and employed controls using Prompt Injection attacks. We derive a comprehensive taxonomy from a computer security perspective to systematically investigate impacts and vulnerabilities.
  arXiv  Detail & Related papers  (2023-02-23T17:14:38Z)
• Threat Assessment in Machine Learning based Systems [12.031113181911627]
  We conduct an empirical study of threats reported against Machine Learning-based systems. The study is based on 89 real-world ML attack scenarios from the MITRE's ATLAS database, the AI Incident Database, and the literature. Results show that convolutional neural networks were one of the most targeted models among the attack scenarios.
  arXiv  Detail & Related papers  (2022-06-30T20:19:50Z)
• Adversarial Machine Learning Threat Analysis in Open Radio Access Networks [37.23982660941893]
  The Open Radio Access Network (O-RAN) is a new, open, adaptive, and intelligent RAN architecture. In this paper, we present a systematic adversarial machine learning threat analysis for the O-RAN.
  arXiv  Detail & Related papers  (2022-01-16T17:01:38Z)
• RoFL: Attestable Robustness for Secure Federated Learning [59.63865074749391]
  Federated Learning allows a large number of clients to train a joint model without the need to share their private data. To ensure the confidentiality of the client updates, Federated Learning systems employ secure aggregation. We present RoFL, a secure Federated Learning system that improves robustness against malicious clients.
  arXiv  Detail & Related papers  (2021-07-07T15:42:49Z)
• Dos and Don'ts of Machine Learning in Computer Security [74.1816306998445]
  Despite great potential, machine learning in security is prone to subtle pitfalls that undermine its performance. We identify common pitfalls in the design, implementation, and evaluation of learning-based security systems. We propose actionable recommendations to support researchers in avoiding or mitigating the pitfalls where possible.
  arXiv  Detail & Related papers  (2020-10-19T13:09:31Z)
• Adversarial Machine Learning: Bayesian Perspectives [0.4915744683251149]
  Adversarial Machine Learning (AML) is emerging as a major field aimed at protecting machine learning (ML) systems against security threats. In certain scenarios there may be adversaries that actively manipulate input data to fool learning systems. This creates a new class of security vulnerabilities that ML systems may face, and a new desirable property called adversarial robustness essential to trust operations.
  arXiv  Detail & Related papers  (2020-03-07T10:30:43Z)
• Enhanced Adversarial Strategically-Timed Attacks against Deep Reinforcement Learning [91.13113161754022]
  We introduce timing-based adversarial strategies against a DRL-based navigation system by jamming in physical noise patterns on the selected time frames. Our experimental results show that the adversarial timing attacks can lead to a significant performance drop.
  arXiv  Detail & Related papers  (2020-02-20T21:39:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.