When AI Meets the Web: Prompt Injection Risks in Third-Party AI Chatbot Plugins

• URL: http://arxiv.org/abs/2511.05797v1
• Date: Sat, 08 Nov 2025 02:02:24 GMT
• Title: When AI Meets the Web: Prompt Injection Risks in Third-Party AI Chatbot Plugins
• Abstract summary: We present the first large-scale study of 17 third-party plugins used by over 10,000 public websites.<n>8 of these plugins fail to enforce the integrity of the conversation history transmitted in network requests.<n>We found that 13% of e-commerce websites have already exposed their chatbots to third-party content.
• Score: 17.279238456399693
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prompt injection attacks pose a critical threat to large language models (LLMs), with prior work focusing on cutting-edge LLM applications like personal copilots. In contrast, simpler LLM applications, such as customer service chatbots, are widespread on the web, yet their security posture and exposure to such attacks remain poorly understood. These applications often rely on third-party chatbot plugins that act as intermediaries to commercial LLM APIs, offering non-expert website builders intuitive ways to customize chatbot behaviors. To bridge this gap, we present the first large-scale study of 17 third-party chatbot plugins used by over 10,000 public websites, uncovering previously unknown prompt injection risks in practice. First, 8 of these plugins (used by 8,000 websites) fail to enforce the integrity of the conversation history transmitted in network requests between the website visitor and the chatbot. This oversight amplifies the impact of direct prompt injection attacks by allowing adversaries to forge conversation histories (including fake system messages), boosting their ability to elicit unintended behavior (e.g., code generation) by 3 to 8x. Second, 15 plugins offer tools, such as web-scraping, to enrich the chatbot's context with website-specific content. However, these tools do not distinguish the website's trusted content (e.g., product descriptions) from untrusted, third-party content (e.g., customer reviews), introducing a risk of indirect prompt injection. Notably, we found that ~13% of e-commerce websites have already exposed their chatbots to third-party content. We systematically evaluate both vulnerabilities through controlled experiments grounded in real-world observations, focusing on factors such as system prompt design and the underlying LLM. Our findings show that many plugins adopt insecure practices that undermine the built-in LLM safeguards.

Related papers

• Plant, Persist, Trigger: Sleeper Attack on Large Language Model Agents [64.80318459073526]
  We show that adversarial content can persist across interactions served by the same agent, making such threats harder to detect and mitigate.<n>Specifically, adversarial content may persist in the agent state, remain dormant across interactions, and later be activated by a benign user query.<n>We formalize this type of safety threat as Sleeper Attack. To evaluate it, we construct a benchmark with 1,896 instances covering six real-world harmful outcomes, three attack strategies, and three agent state targets.
  arXiv  Detail & Related papers  (2026-05-27T09:25:37Z)
• An Empirical Study of Privacy Leakage Chains via Prompt Injection in Black-Box Chatbot Environments [3.275342475988451]
  This paper studies a privacy-leakage attack chain based on indirect prompt injection.<n>We first analyze how an attacker can hijack an agent's intended task by crafting external content that appears benign to the victim.<n>We then evaluate a new prompt-language technique, called exemplification, which uses a bridge in the external content to reframe the user prompt and the benign beginning of the retrieved page.
  arXiv  Detail & Related papers  (2026-05-18T09:38:18Z)
• Indirect Prompt Injection in the Wild: An Empirical Study of Prevalence, Techniques, and Objectives [15.03910974112065]
  LLMs are increasingly integrated into systems that browse, retrieve, summarize, and act on web content.<n>This enables site owners, contributors, and adversaries to embed instructions directly in web resources.<n>We present one of the first large-scale empirical analyses of indirect prompt injections in webpages and HTTP responses.
  arXiv  Detail & Related papers  (2026-04-29T21:09:21Z)
• WebAgentGuard: A Reasoning-Driven Guard Model for Detecting Prompt Injection Attacks in Web Agents [117.65855863464863]
  Web agents are highly vulnerable to prompt injection attacks.<n>Existing defenses, including system prompt defenses and direct fine-tuning of agents, have shown limited effectiveness.<n>We introduce WebAgentGuard, a reasoning-driven, multimodal guard model for prompt injection detection.
  arXiv  Detail & Related papers  (2026-04-14T04:50:35Z)
• It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agents [52.81924177620322]
  Web-based agents powered by large language models are increasingly used for tasks such as email management or professional networking.<n>Their reliance on dynamic web content makes them vulnerable to prompt injection attacks: adversarial instructions hidden in interface elements that persuade the agent to divert from its original task.<n>We introduce the Task-Redirecting Agent Persuasion Benchmark (TRAP), an evaluation for studying how persuasion techniques misguide autonomous web agents on realistic tasks.
  arXiv  Detail & Related papers  (2025-12-29T01:09:10Z)
• ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agents [15.773168171609754]
  We introduce ChatInject, an attack that formats malicious payloads to mimic native chat templates.<n>We develop a persuasion-driven Multi-turn variant that primes the agent across conversational turns to accept and execute otherwise suspicious actions.<n>ChatInject achieves significantly higher average attack success rates than traditional prompt injection methods.
  arXiv  Detail & Related papers  (2025-09-26T18:38:07Z)
• Ask ChatGPT: Caveats and Mitigations for Individual Users of AI Chatbots [10.977907906989342]
  ChatGPT and other Large Language Model (LLM)-based AI chatbots become increasingly integrated into individuals' daily lives.<n>What concerns and risks do these systems pose for individual users?<n>What potential harms might they cause, and how can these be mitigated?
  arXiv  Detail & Related papers  (2025-08-14T01:40:13Z)
• Manipulating LLM Web Agents with Indirect Prompt Injection Attack via HTML Accessibility Tree [8.511846002129522]
  We show that adversaries can embed universal adversarial triggers in webpage HTML to hijack agent behavior.<n>Our system demonstrates high success rates across real websites in both targeted and general attacks.
  arXiv  Detail & Related papers  (2025-07-20T03:10:13Z)
• OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents [60.78202583483591]
  We introduce OS-Harm, a new benchmark for measuring safety of computer use agents.<n> OS-Harm is built on top of the OSWorld environment and aims to test models across three categories of harm: deliberate user misuse, prompt injection attacks, and model misbehavior.<n>We evaluate computer use agents based on a range of frontier models and provide insights into their safety.
  arXiv  Detail & Related papers  (2025-06-17T17:59:31Z)
• SafeChat: A Framework for Building Trustworthy Collaborative Assistants and a Case Study of its Usefulness [4.896226014796392]
  We introduce SafeChat, a general architecture for building safe and trustworthy chatbots.<n>Key features of SafeChat include: (a) safety, with a domain-agnostic design where responses are grounded and traceable to approved sources (provenance); (b) usability, with automatic extractive summarization of long responses, traceable to their sources; and (c) fast, scalable development, including a CSV-driven workflow, automated testing, and integration with various devices.
  arXiv  Detail & Related papers  (2025-04-08T19:16:43Z)
• Human-Readable Adversarial Prompts: An Investigation into LLM Vulnerabilities Using Situational Context [45.821481786228226]
  We show that situation-driven adversarial full-prompts that leverage situational context are effective but much harder to detect.<n>We developed attacks that use movie scripts as situational contextual frameworks.<n>We enhanced the AdvPrompter framework with p-nucleus sampling to generate diverse human-readable adversarial texts.
  arXiv  Detail & Related papers  (2024-12-20T21:43:52Z)
• SecAlign: Defending Against Prompt Injection with Preference Optimization [52.48001255555192]
  Adversarial prompts can be injected into external data sources to override the system's intended instruction and execute a malicious instruction.<n>We propose a new defense called SecAlign based on the technique of preference optimization.<n>Our method reduces the success rates of various prompt injections to 10%, even against attacks much more sophisticated than ones seen during training.
  arXiv  Detail & Related papers  (2024-10-07T19:34:35Z)
• SPML: A DSL for Defending Language Models Against Prompt Attacks [4.511923587827302]
  System Prompt Meta Language (SPML) is a domain-specific language for refining prompts and monitoring inputs to large language models (LLMs) SPML actively checks attack prompts, ensuring user inputs align with definitions to prevent malicious execution on the LLM backbone, optimizing costs. We introduce a groundbreaking benchmark with 1.8k system prompts and 20k user inputs, offering the inaugural language and benchmark for chatbots definition evaluation.
  arXiv  Detail & Related papers  (2024-02-19T00:53:48Z)
• Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game [86.66627242073724]
  This paper presents a dataset of over 126,000 prompt injection attacks and 46,000 prompt-based "defenses" against prompt injection. To the best of our knowledge, this is currently the largest dataset of human-generated adversarial examples for instruction-following LLMs. We also use the dataset to create a benchmark for resistance to two types of prompt injection, which we refer to as prompt extraction and prompt hijacking.
  arXiv  Detail & Related papers  (2023-11-02T06:13:36Z)
• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection [64.67495502772866]
  Large Language Models (LLMs) are increasingly being integrated into various applications. We show how attackers can override original instructions and employed controls using Prompt Injection attacks. We derive a comprehensive taxonomy from a computer security perspective to systematically investigate impacts and vulnerabilities.
  arXiv  Detail & Related papers  (2023-02-23T17:14:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.