On the (In)Security of LLM App Stores

• URL: http://arxiv.org/abs/2407.08422v2
• Date: Mon, 29 Jul 2024 11:18:57 GMT
• Title: On the (In)Security of LLM App Stores
• Authors: Xinyi Hou, Yanjie Zhao, Haoyu Wang,
• Abstract summary: In this study, we propose a three-layer concern framework to identify the potential security risks of LLM apps. Our research integrates static and dynamic analysis, the development of a large-scale toxic word dictionary, and automated monitoring tools. We uncovered that 15,146 apps had misleading descriptions, 1,366 collected sensitive personal information against their privacy policies, and 15,996 generated harmful content.
• Score: 5.667013605202579
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM app stores have seen rapid growth, leading to the proliferation of numerous custom LLM apps. However, this expansion raises security concerns. In this study, we propose a three-layer concern framework to identify the potential security risks of LLM apps, i.e., LLM apps with abusive potential, LLM apps with malicious intent, and LLM apps with exploitable vulnerabilities. Over five months, we collected 786,036 LLM apps from six major app stores: GPT Store, FlowGPT, Poe, Coze, Cici, and Character.AI. Our research integrates static and dynamic analysis, the development of a large-scale toxic word dictionary (i.e., ToxicDict) comprising over 31,783 entries, and automated monitoring tools to identify and mitigate threats. We uncovered that 15,146 apps had misleading descriptions, 1,366 collected sensitive personal information against their privacy policies, and 15,996 generated harmful content such as hate speech, self-harm, extremism, etc. Additionally, we evaluated the potential for LLM apps to facilitate malicious activities, finding that 616 apps could be used for malware generation, phishing, etc. Our findings highlight the urgent need for robust regulatory frameworks and enhanced enforcement mechanisms.

Related papers

• AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents [84.96249955105777]
  LLM agents may pose a greater risk if misused, but their robustness remains underexplored. We propose a new benchmark called AgentHarm to facilitate research on LLM agent misuse. We find leading LLMs are surprisingly compliant with malicious agent requests without jailbreaking.
  arXiv  Detail & Related papers  (2024-10-11T17:39:22Z)
• Exploring LLMs for Malware Detection: Review, Framework Design, and Countermeasure Approaches [0.24578723416255752]
  The rising use of Large Language Models to create and disseminate malware poses a significant cybersecurity challenge. This paper provides a comprehensive overview of LLMs and their role in malware detection from diverse sources. We examine five specific applications of LLMs: Malware honeypots, identification of text-based threats, code analysis for detecting malicious intent, trend analysis of malware, and detection of non-standard disguised malware.
  arXiv  Detail & Related papers  (2024-09-11T19:33:44Z)
• MaPPing Your Model: Assessing the Impact of Adversarial Attacks on LLM-based Programming Assistants [14.947665219536708]
  We introduce the Malicious Programming Prompt (MaPP) attack, in which an attacker adds a small amount of text to a prompt for a programming task. We show that our prompt strategy can cause an LLM to add vulnerabilities while continuing to write otherwise correct code.
  arXiv  Detail & Related papers  (2024-07-12T22:30:35Z)
• Human-Imperceptible Retrieval Poisoning Attacks in LLM-Powered Applications [10.06789804722156]
  We reveal a new threat to LLM-powered applications, termed retrieval poisoning, where attackers can guide the application to yield malicious responses during the RAG process. Our preliminary experiments indicate that attackers can mislead LLMs with an 88.33% success rate, and achieve a 66.67% success rate in the real-world application.
  arXiv  Detail & Related papers  (2024-04-26T07:11:18Z)
• Prompt Leakage effect and defense strategies for multi-turn LLM interactions [95.33778028192593]
  Leakage of system prompts may compromise intellectual property and act as adversarial reconnaissance for an attacker. We design a unique threat model which leverages the LLM sycophancy effect and elevates the average attack success rate (ASR) from 17.7% to 86.2% in a multi-turn setting. We measure the mitigation effect of 7 black-box defense strategies, along with finetuning an open-source model to defend against leakage attempts.
  arXiv  Detail & Related papers  (2024-04-24T23:39:58Z)
• Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation [98.02846901473697]
  We propose ECSO (Eyes Closed, Safety On), a training-free protecting approach that exploits the inherent safety awareness of MLLMs. ECSO generates safer responses via adaptively transforming unsafe images into texts to activate the intrinsic safety mechanism of pre-aligned LLMs.
  arXiv  Detail & Related papers  (2024-03-14T17:03:04Z)
• Rethinking Jailbreaking through the Lens of Representation Engineering [45.70565305714579]
  The recent surge in jailbreaking methods has revealed the vulnerability of Large Language Models (LLMs) to malicious inputs. This study investigates the vulnerability of safety-aligned LLMs by uncovering specific activity patterns.
  arXiv  Detail & Related papers  (2024-01-12T00:50:04Z)
• A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly [21.536079040559517]
  Large Language Models (LLMs) have revolutionized natural language understanding and generation. This paper explores the intersection of LLMs with security and privacy.
  arXiv  Detail & Related papers  (2023-12-04T16:25:18Z)
• Identifying and Mitigating Vulnerabilities in LLM-Integrated Applications [37.316238236750415]
  Large language models (LLMs) are increasingly deployed as the service backend for LLM-integrated applications. In this work, we consider a setup where the user and LLM interact via an LLM-integrated application in the middle. We identify potential vulnerabilities that can originate from the malicious application developer or from an outsider threat. We develop a lightweight, threat-agnostic defense that mitigates both insider and outsider threats.
  arXiv  Detail & Related papers  (2023-11-07T20:13:05Z)
• Making Harmful Behaviors Unlearnable for Large Language Models [50.44915524846857]
  Large language models (LLMs) have shown great potential as general-purpose AI assistants in various domains. LLMs can be easily fine-tuned into harmful assistants as the fine-tuning data often contains implicit or explicit harmful content. This paper proposes a controllable training framework that makes harmful behaviors unlearnable during the fine-tuning process.
  arXiv  Detail & Related papers  (2023-11-02T09:18:21Z)
• Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs [59.596335292426105]
  This paper collects the first open-source dataset to evaluate safeguards in large language models. We train several BERT-like classifiers to achieve results comparable with GPT-4 on automatic safety evaluation.
  arXiv  Detail & Related papers  (2023-08-25T14:02:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.