Invitation Is All You Need! Promptware Attacks Against LLM-Powered Assistants in Production Are Practical and Dangerous

• URL: http://arxiv.org/abs/2508.12175v1
• Date: Sat, 16 Aug 2025 22:56:51 GMT
• Title: Invitation Is All You Need! Promptware Attacks Against LLM-Powered Assistants in Production Are Practical and Dangerous
• Abstract summary: We propose a novel Threat Analysis and Risk Assessment framework to assess Promptware risks for end users.<n>We demonstrate 14 attack scenarios applied against Gemini-powered assistants across five identified threat classes.<n>Our TARA reveals that 73% of the analyzed threats pose High-Critical risk to end users.
• Score: 4.172921042226532
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The growing integration of LLMs into applications has introduced new security risks, notably known as Promptware - maliciously engineered prompts designed to manipulate LLMs to compromise the CIA triad of these applications. While prior research warned about a potential shift in the threat landscape for LLM-powered applications, the risk posed by Promptware is frequently perceived as low. In this paper, we investigate the risk Promptware poses to users of Gemini-powered assistants (web application, mobile application, and Google Assistant). We propose a novel Threat Analysis and Risk Assessment (TARA) framework to assess Promptware risks for end users. Our analysis focuses on a new variant of Promptware called Targeted Promptware Attacks, which leverage indirect prompt injection via common user interactions such as emails, calendar invitations, and shared documents. We demonstrate 14 attack scenarios applied against Gemini-powered assistants across five identified threat classes: Short-term Context Poisoning, Permanent Memory Poisoning, Tool Misuse, Automatic Agent Invocation, and Automatic App Invocation. These attacks highlight both digital and physical consequences, including spamming, phishing, disinformation campaigns, data exfiltration, unapproved user video streaming, and control of home automation devices. We reveal Promptware's potential for on-device lateral movement, escaping the boundaries of the LLM-powered application, to trigger malicious actions using a device's applications. Our TARA reveals that 73% of the analyzed threats pose High-Critical risk to end users. We discuss mitigations and reassess the risk (in response to deployed mitigations) and show that the risk could be reduced significantly to Very Low-Medium. We disclosed our findings to Google, which deployed dedicated mitigations.

Related papers

• Beware of Agentic Botnets: Scalable Untargeted Promptware Attacks via Universal and Transferable Adversarial HalluSquatting [10.894389368294968]
  We show that the inherent tendency of LLMs to hallucinate resource identifiers can be exploited to amplify untargeted promptware attacks.<n>We introduce adversarial hallucination squatting, a technique in which attackers identify trending resources and register them to host adversarial prompts.<n>By leveraging the predictability and transferability of hallucinations across foundational LLMs and to application layers, adversaries can significantly amplify the reach of untargeted promptware.
  arXiv  Detail & Related papers  (2026-07-08T14:02:14Z)
• Prompt-in-Content Attacks: Exploiting Uploaded Inputs to Hijack LLM Behavior [3.715687323488774]
  Large Language Models (LLMs) are widely deployed in applications that accept user-submitted content.<n>In this paper, we identify a new class of attacks, prompt in content injection, where adversarial instructions are embedded in seemingly benign inputs.
  arXiv  Detail & Related papers  (2025-08-25T05:20:11Z)
• AutoAdv: Automated Adversarial Prompting for Multi-Turn Jailbreaking of Large Language Models [0.0]
  Large Language Models (LLMs) continue to exhibit vulnerabilities to jailbreaking attacks.<n>We present AutoAdv, a novel framework that automates adversarial prompt generation.<n>We show that our attacks achieve jailbreak success rates of up to 86% for harmful content generation.
  arXiv  Detail & Related papers  (2025-04-18T08:38:56Z)
• Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks [88.84977282952602]
  A high volume of recent ML security literature focuses on attacks against aligned large language models (LLMs)<n>In this paper, we analyze security and privacy vulnerabilities that are unique to LLM agents.<n>We conduct a series of illustrative attacks on popular open-source and commercial agents, demonstrating the immediate practical implications of their vulnerabilities.
  arXiv  Detail & Related papers  (2025-02-12T17:19:36Z)
• SecAlign: Defending Against Prompt Injection with Preference Optimization [52.48001255555192]
  Adversarial prompts can be injected into external data sources to override the system's intended instruction and execute a malicious instruction.<n>We propose a new defense called SecAlign based on the technique of preference optimization.<n>Our method reduces the success rates of various prompt injections to 10%, even against attacks much more sophisticated than ones seen during training.
  arXiv  Detail & Related papers  (2024-10-07T19:34:35Z)
• Human-Interpretable Adversarial Prompt Attack on Large Language Models with Situational Context [49.13497493053742]
  This research explores converting a nonsensical suffix attack into a sensible prompt via a situation-driven contextual re-writing. We combine an independent, meaningful adversarial insertion and situations derived from movies to check if this can trick an LLM. Our approach demonstrates that a successful situation-driven attack can be executed on both open-source and proprietary LLMs.
  arXiv  Detail & Related papers  (2024-07-19T19:47:26Z)
• Prompt Leakage effect and defense strategies for multi-turn LLM interactions [95.33778028192593]
  Leakage of system prompts may compromise intellectual property and act as adversarial reconnaissance for an attacker. We design a unique threat model which leverages the LLM sycophancy effect and elevates the average attack success rate (ASR) from 17.7% to 86.2% in a multi-turn setting. We measure the mitigation effect of 7 black-box defense strategies, along with finetuning an open-source model to defend against leakage attempts.
  arXiv  Detail & Related papers  (2024-04-24T23:39:58Z)
• AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs [51.217126257318924]
  Large Language Models (LLMs) are vulnerable to jailbreaking attacks that lead to generation of inappropriate or harmful content.<n>We present a novel method that uses another LLM, called AdvPrompter, to generate human-readable adversarial prompts in seconds.
  arXiv  Detail & Related papers  (2024-04-21T22:18:13Z)
• Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models [79.0183835295533]
  We introduce the first benchmark for indirect prompt injection attacks, named BIPIA, to assess the risk of such vulnerabilities.<n>Our analysis identifies two key factors contributing to their success: LLMs' inability to distinguish between informational context and actionable instructions, and their lack of awareness in avoiding the execution of instructions within external content.<n>We propose two novel defense mechanisms-boundary awareness and explicit reminder-to address these vulnerabilities in both black-box and white-box settings.
  arXiv  Detail & Related papers  (2023-12-21T01:08:39Z)
• Identifying and Mitigating Vulnerabilities in LLM-Integrated Applications [37.316238236750415]
  Large language models (LLMs) are increasingly deployed as the service backend for LLM-integrated applications. In this work, we consider a setup where the user and LLM interact via an LLM-integrated application in the middle. We identify potential vulnerabilities that can originate from the malicious application developer or from an outsider threat. We develop a lightweight, threat-agnostic defense that mitigates both insider and outsider threats.
  arXiv  Detail & Related papers  (2023-11-07T20:13:05Z)
• Prompt Injection attack against LLM-integrated Applications [37.86878788874201]
  This study deconstructs the complexities and implications of prompt injection attacks on actual LLM-integrated applications. We formulate HouYi, a novel black-box prompt injection attack technique, which draws inspiration from traditional web injection attacks. We deploy HouYi on 36 actual LLM-integrated applications and discern 31 applications susceptible to prompt injection.
  arXiv  Detail & Related papers  (2023-06-08T18:43:11Z)
• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection [64.67495502772866]
  Large Language Models (LLMs) are increasingly being integrated into various applications. We show how attackers can override original instructions and employed controls using Prompt Injection attacks. We derive a comprehensive taxonomy from a computer security perspective to systematically investigate impacts and vulnerabilities.
  arXiv  Detail & Related papers  (2023-02-23T17:14:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.