A Large-Scale Empirical Analysis of Custom GPTs' Vulnerabilities in the OpenAI Ecosystem

• URL: http://arxiv.org/abs/2505.08148v1
• Date: Tue, 13 May 2025 00:51:07 GMT
• Title: A Large-Scale Empirical Analysis of Custom GPTs' Vulnerabilities in the OpenAI Ecosystem
• Authors: Sunday Oyinlola Ogundoyin, Muhammad Ikram, Hassan Jameel Asghar, Benjamin Zi Hao Zhao, Dali Kaafar,
• Abstract summary: We analyze 14,904 custom GPTs to assess their susceptibility to seven exploitable threats.<n>Our findings reveal that over 95% of custom GPTs lack adequate security protections.<n>These results highlight the urgent need for enhanced security measures and stricter content moderation.
• Score: 5.455788617334495
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Millions of users leverage generative pretrained transformer (GPT)-based language models developed by leading model providers for a wide range of tasks. To support enhanced user interaction and customization, many platforms-such as OpenAI-now enable developers to create and publish tailored model instances, known as custom GPTs, via dedicated repositories or application stores. These custom GPTs empower users to browse and interact with specialized applications designed to meet specific needs. However, as custom GPTs see growing adoption, concerns regarding their security vulnerabilities have intensified. Existing research on these vulnerabilities remains largely theoretical, often lacking empirical, large-scale, and statistically rigorous assessments of associated risks. In this study, we analyze 14,904 custom GPTs to assess their susceptibility to seven exploitable threats, such as roleplay-based attacks, system prompt leakage, phishing content generation, and malicious code synthesis, across various categories and popularity tiers within the OpenAI marketplace. We introduce a multi-metric ranking system to examine the relationship between a custom GPT's popularity and its associated security risks. Our findings reveal that over 95% of custom GPTs lack adequate security protections. The most prevalent vulnerabilities include roleplay-based vulnerabilities (96.51%), system prompt leakage (92.20%), and phishing (91.22%). Furthermore, we demonstrate that OpenAI's foundational models exhibit inherent security weaknesses, which are often inherited or amplified in custom GPTs. These results highlight the urgent need for enhanced security measures and stricter content moderation to ensure the safe deployment of GPT-based applications.

Related papers

• Security Assessment of DeepSeek and GPT Series Models against Jailbreak Attacks [9.277492743469235]
  We present the first systematic jailbreak evaluation of DeepSeek-series models.<n>We compare them with GPT-3.5 and GPT-4 using the HarmBench benchmark.
  arXiv  Detail & Related papers  (2025-06-23T11:53:31Z)
• Privacy and Security Threat for OpenAI GPTs [0.0]
  Since OpenAI's release in November 2023, over 3 million custom GPTs have been created.<n>For developers, instruction leaking attacks threaten the intellectual property of instructions in custom GPTs.<n>For users, unwanted data access behavior by custom GPTs or integrated third-party services raises significant privacy concerns.
  arXiv  Detail & Related papers  (2025-06-04T14:58:29Z)
• Towards Trustworthy GUI Agents: A Survey [64.6445117343499]
  This survey examines the trustworthiness of GUI agents in five critical dimensions.<n>We identify major challenges such as vulnerability to adversarial attacks, cascading failure modes in sequential decision-making.<n>As GUI agents become more widespread, establishing robust safety standards and responsible development practices is essential.
  arXiv  Detail & Related papers  (2025-03-30T13:26:00Z)
• In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AI [93.33036653316591]
  We call for three interventions to advance system safety.<n>First, we propose using standardized AI flaw reports and rules of engagement for researchers.<n>Second, we propose GPAI system providers adopt broadly-scoped flaw disclosure programs.<n>Third, we advocate for the development of improved infrastructure to coordinate distribution of flaw reports.
  arXiv  Detail & Related papers  (2025-03-21T05:09:46Z)
• AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons [62.374792825813394]
  This paper introduces AILuminate v1.0, the first comprehensive industry-standard benchmark for assessing AI-product risk and reliability.<n>The benchmark evaluates an AI system's resistance to prompts designed to elicit dangerous, illegal, or undesirable behavior in 12 hazard categories.
  arXiv  Detail & Related papers  (2025-02-19T05:58:52Z)
• Towards Safer Chatbots: A Framework for Policy Compliance Evaluation of Custom GPTs [7.687215328455751]
  We present a framework for the automated evaluation of Custom GPTs against OpenAI's usage policies.<n>We evaluate it through a large-scale study with 782 Custom GPTs across three categories: Romantic, Cybersecurity, and Academic GPTs.<n>The results reveal that 58.7% of the analyzed models exhibit indications of non-compliance, exposing weaknesses in the GPT store's review and approval processes.
  arXiv  Detail & Related papers  (2025-02-03T15:19:28Z)
• Rethinking the Vulnerabilities of Face Recognition Systems:From a Practical Perspective [53.24281798458074]
  Face Recognition Systems (FRS) have increasingly integrated into critical applications, including surveillance and user authentication. Recent studies have revealed vulnerabilities in FRS to adversarial (e.g., adversarial patch attacks) and backdoor attacks (e.g., training data poisoning)
  arXiv  Detail & Related papers  (2024-05-21T13:34:23Z)
• Opening A Pandora's Box: Things You Should Know in the Era of Custom GPTs [27.97654690288698]
  We conduct a comprehensive analysis of the security and privacy issues arising from the custom GPT platform by OpenAI. Our systematic examination categorizes potential attack scenarios into three threat models based on the role of the malicious actor. We identify 26 potential attack vectors, with 19 being partially or fully validated in real-world settings.
  arXiv  Detail & Related papers  (2023-12-31T16:49:12Z)
• Assessing Prompt Injection Risks in 200+ Custom GPTs [21.86130076843285]
  This study reveals a significant security vulnerability inherent in user-customized GPTs: prompt injection attacks. Through prompt injection, an adversary can not only extract the customized system prompts but also access the uploaded files. This paper provides a first-hand analysis of the prompt injection, alongside the evaluation of the possible mitigation of such attacks.
  arXiv  Detail & Related papers  (2023-11-20T04:56:46Z)
• DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models [92.6951708781736]
  This work proposes a comprehensive trustworthiness evaluation for large language models with a focus on GPT-4 and GPT-3.5. We find that GPT models can be easily misled to generate toxic and biased outputs and leak private information. Our work illustrates a comprehensive trustworthiness evaluation of GPT models and sheds light on the trustworthiness gaps.
  arXiv  Detail & Related papers  (2023-06-20T17:24:23Z)
• G$^2$uardFL: Safeguarding Federated Learning Against Backdoor Attacks through Attributed Client Graph Clustering [116.4277292854053]
  Federated Learning (FL) offers collaborative model training without data sharing. FL is vulnerable to backdoor attacks, where poisoned model weights lead to compromised system integrity. We present G$2$uardFL, a protective framework that reinterprets the identification of malicious clients as an attributed graph clustering problem.
  arXiv  Detail & Related papers  (2023-06-08T07:15:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.