Privacy and Security Threat for OpenAI GPTs

• URL: http://arxiv.org/abs/2506.04036v1
• Date: Wed, 04 Jun 2025 14:58:29 GMT
• Title: Privacy and Security Threat for OpenAI GPTs
• Authors: Wei Wenying, Zhao Kaifa, Xue Lei, Fan Ming,
• Abstract summary: Since OpenAI's release in November 2023, over 3 million custom GPTs have been created.<n>For developers, instruction leaking attacks threaten the intellectual property of instructions in custom GPTs.<n>For users, unwanted data access behavior by custom GPTs or integrated third-party services raises significant privacy concerns.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) demonstrate powerful information handling capabilities and are widely integrated into chatbot applications. OpenAI provides a platform for developers to construct custom GPTs, extending ChatGPT's functions and integrating external services. Since its release in November 2023, over 3 million custom GPTs have been created. However, such a vast ecosystem also conceals security and privacy threats. For developers, instruction leaking attacks threaten the intellectual property of instructions in custom GPTs through carefully crafted adversarial prompts. For users, unwanted data access behavior by custom GPTs or integrated third-party services raises significant privacy concerns. To systematically evaluate the scope of threats in real-world LLM applications, we develop three phases instruction leaking attacks target GPTs with different defense level. Our widespread experiments on 10,000 real-world custom GPTs reveal that over 98.8% of GPTs are vulnerable to instruction leaking attacks via one or more adversarial prompts, and half of the remaining GPTs can also be attacked through multiround conversations. We also developed a framework to assess the effectiveness of defensive strategies and identify unwanted behaviors in custom GPTs. Our findings show that 77.5% of custom GPTs with defense strategies are vulnerable to basic instruction leaking attacks. Additionally, we reveal that 738 custom GPTs collect user conversational information, and identified 8 GPTs exhibiting data access behaviors that are unnecessary for their intended functionalities. Our findings raise awareness among GPT developers about the importance of integrating specific defensive strategies in their instructions and highlight users' concerns about data privacy when using LLM-based applications.

Related papers

• Tracking GPTs Third Party Service: Automation, Analysis, and Insights [9.269295824340858]
  GPTs-ThirdSpy is an automated framework designed to extract privacy settings of GPTs.<n> GPTs-ThirdSpy provides academic researchers with real-time, reliable metadata on third-party services used by GPTs.
  arXiv  Detail & Related papers  (2025-06-18T13:41:14Z)
• A Large-Scale Empirical Analysis of Custom GPTs' Vulnerabilities in the OpenAI Ecosystem [5.455788617334495]
  We analyze 14,904 custom GPTs to assess their susceptibility to seven exploitable threats.<n>Our findings reveal that over 95% of custom GPTs lack adequate security protections.<n>These results highlight the urgent need for enhanced security measures and stricter content moderation.
  arXiv  Detail & Related papers  (2025-05-13T00:51:07Z)
• Towards Safer Chatbots: A Framework for Policy Compliance Evaluation of Custom GPTs [7.687215328455751]
  We present a framework for the automated evaluation of Custom GPTs against OpenAI's usage policies.<n>We evaluate it through a large-scale study with 782 Custom GPTs across three categories: Romantic, Cybersecurity, and Academic GPTs.<n>The results reveal that 58.7% of the analyzed models exhibit indications of non-compliance, exposing weaknesses in the GPT store's review and approval processes.
  arXiv  Detail & Related papers  (2025-02-03T15:19:28Z)
• PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action [54.11479432110771]
  PrivacyLens is a novel framework designed to extend privacy-sensitive seeds into expressive vignettes and further into agent trajectories.<n>We instantiate PrivacyLens with a collection of privacy norms grounded in privacy literature and crowdsourced seeds.<n>State-of-the-art LMs, like GPT-4 and Llama-3-70B, leak sensitive information in 25.68% and 38.69% of cases, even when prompted with privacy-enhancing instructions.
  arXiv  Detail & Related papers  (2024-08-29T17:58:38Z)
• Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuning [62.224804688233]
  differential privacy (DP) offers a promising solution by ensuring models are 'almost indistinguishable' with or without any particular privacy unit. We study user-level DP motivated by applications where it necessary to ensure uniform privacy protection across users.
  arXiv  Detail & Related papers  (2024-06-20T13:54:32Z)
• A First Look at GPT Apps: Landscape and Vulnerability [14.869850673247631]
  This study focuses on two GPT app stores: textitGPTStore.AI and the official textitOpenAI GPT Store.<n>Specifically, we develop two automated tools and a TriLevel configuration extraction strategy to efficiently gather metadata for all GPT apps across these two stores.<n>Our extensive analysis reveals: (1) the user enthusiasm for GPT apps consistently rises, whereas creator interest plateaus within three months of GPTs' launch; (2) nearly 90%% system prompts can be easily accessed due to widespread failure to secure GPT app configurations.
  arXiv  Detail & Related papers  (2024-02-23T05:30:32Z)
• Reconstruct Your Previous Conversations! Comprehensively Investigating Privacy Leakage Risks in Conversations with GPT Models [20.92843974858305]
  GPT models are increasingly being used for task optimization. In this paper, we introduce a straightforward yet potent Conversation Reconstruction Attack. We present two advanced attacks targeting improved reconstruction of past conversations.
  arXiv  Detail & Related papers  (2024-02-05T13:18:42Z)
• Opening A Pandora's Box: Things You Should Know in the Era of Custom GPTs [27.97654690288698]
  We conduct a comprehensive analysis of the security and privacy issues arising from the custom GPT platform by OpenAI. Our systematic examination categorizes potential attack scenarios into three threat models based on the role of the malicious actor. We identify 26 potential attack vectors, with 19 being partially or fully validated in real-world settings.
  arXiv  Detail & Related papers  (2023-12-31T16:49:12Z)
• Exploring ChatGPT's Capabilities on Vulnerability Management [56.4403395100589]
  We explore ChatGPT's capabilities on 6 tasks involving the complete vulnerability management process with a large-scale dataset containing 70,346 samples. One notable example is ChatGPT's proficiency in tasks like generating titles for software bug reports. Our findings reveal the difficulties encountered by ChatGPT and shed light on promising future directions.
  arXiv  Detail & Related papers  (2023-11-11T11:01:13Z)
• DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models [92.6951708781736]
  This work proposes a comprehensive trustworthiness evaluation for large language models with a focus on GPT-4 and GPT-3.5. We find that GPT models can be easily misled to generate toxic and biased outputs and leak private information. Our work illustrates a comprehensive trustworthiness evaluation of GPT models and sheds light on the trustworthiness gaps.
  arXiv  Detail & Related papers  (2023-06-20T17:24:23Z)
• ChatGPT for Us: Preserving Data Privacy in ChatGPT via Dialogue Text Ambiguation to Expand Mental Health Care Delivery [52.73936514734762]
  ChatGPT has gained popularity for its ability to generate human-like dialogue. Data-sensitive domains face challenges in using ChatGPT due to privacy and data-ownership concerns. We propose a text ambiguation framework that preserves user privacy.
  arXiv  Detail & Related papers  (2023-05-19T02:09:52Z)
• Multi-step Jailbreaking Privacy Attacks on ChatGPT [47.10284364632862]
  We study the privacy threats from OpenAI's ChatGPT and the New Bing enhanced by ChatGPT. We conduct extensive experiments to support our claims and discuss LLMs' privacy implications.
  arXiv  Detail & Related papers  (2023-04-11T13:05:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.