A First Look at GPT Apps: Landscape and Vulnerability

• URL: http://arxiv.org/abs/2402.15105v3
• Date: Wed, 27 Nov 2024 07:26:34 GMT
• Title: A First Look at GPT Apps: Landscape and Vulnerability
• Authors: Zejun Zhang, Li Zhang, Xin Yuan, Anlan Zhang, Mengwei Xu, Feng Qian,
• Abstract summary: This study focuses on two GPT app stores: textitGPTStore.AI and the official textitOpenAI GPT Store. Specifically, we develop two automated tools and a TriLevel configuration extraction strategy to efficiently gather metadata for all GPT apps across these two stores. Our extensive analysis reveals: (1) the user enthusiasm for GPT apps consistently rises, whereas creator interest plateaus within three months of GPTs' launch; (2) nearly 90%% system prompts can be easily accessed due to widespread failure to secure GPT app configurations.
• Score: 14.869850673247631
• License:
• Abstract: Following OpenAI's introduction of GPTs, a surge in GPT apps has led to the launch of dedicated LLM app stores. Nevertheless, given its debut, there is a lack of sufficient understanding of this new ecosystem. To fill this gap, this paper presents a first comprehensive longitudinal (5-month) study of the evolution, landscape, and vulnerability of the emerging LLM app ecosystem, focusing on two GPT app stores: \textit{GPTStore.AI} and the official \textit{OpenAI GPT Store}. Specifically, we develop two automated tools and a TriLevel configuration extraction strategy to efficiently gather metadata (\ie names, creators, descriptions, \etc) and user feedback for all GPT apps across these two stores, as well as configurations (\ie system prompts, knowledge files, and APIs) for the top 10,000 popular apps. Our extensive analysis reveals: (1) the user enthusiasm for GPT apps consistently rises, whereas creator interest plateaus within three months of GPTs' launch; (2) nearly 90\% system prompts can be easily accessed due to widespread failure to secure GPT app configurations, leading to considerable plagiarism and duplication among apps. Our findings highlight the necessity of enhancing the LLM app ecosystem by the app stores, creators, and users.

Related papers

• Towards Safer Chatbots: A Framework for Policy Compliance Evaluation of Custom GPTs [7.687215328455751]
  We present a framework for the automated evaluation of Custom GPTs against OpenAI's usage policies. We evaluate it through a large-scale study with 782 Custom GPTs across three categories: Romantic, Cybersecurity, and Academic GPTs. The results reveal that 58.7% of the analyzed models exhibit indications of non-compliance, exposing weaknesses in the GPT store's review and approval processes.
  arXiv  Detail & Related papers  (2025-02-03T15:19:28Z)
• Exploring ChatGPT App Ecosystem: Distribution, Deployment and Security [3.0924093890016904]
  ChatGPT has enabled third-party developers to create plugins to expand ChatGPT's capabilities. We conduct the first comprehensive study of the ChatGPT app ecosystem, aiming to illuminate its landscape for our research community. We uncover an uneven distribution of functionality among ChatGPT plugins, highlighting prevalent and emerging topics.
  arXiv  Detail & Related papers  (2024-08-26T15:31:58Z)
• GPTZoo: A Large-scale Dataset of GPTs for the Research Community [5.1875389249043415]
  GPTZoo is a large-scale dataset comprising 730,420 GPT instances. Each instance includes rich metadata with 21 attributes describing its characteristics, as well as instructions, knowledge files, and third-party services utilized during its development.
  arXiv  Detail & Related papers  (2024-05-24T15:17:03Z)
• MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigation [73.81268591484198]
  Embodied agents equipped with GPT have exhibited extraordinary decision-making and generalization abilities across various tasks. We present a novel map-guided GPT-based agent, dubbed MapGPT, which introduces an online linguistic-formed map to encourage global exploration. Benefiting from this design, we propose an adaptive planning mechanism to assist the agent in performing multi-step path planning based on a map.
  arXiv  Detail & Related papers  (2024-01-14T15:34:48Z)
• Opening A Pandora's Box: Things You Should Know in the Era of Custom GPTs [27.97654690288698]
  We conduct a comprehensive analysis of the security and privacy issues arising from the custom GPT platform by OpenAI. Our systematic examination categorizes potential attack scenarios into three threat models based on the role of the malicious actor. We identify 26 potential attack vectors, with 19 being partially or fully validated in real-world settings.
  arXiv  Detail & Related papers  (2023-12-31T16:49:12Z)
• ChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up? [71.12709925152784]
  ChatGPT has brought a seismic shift in the entire landscape of AI. It showed that a model could answer human questions and follow instructions on a broad panel of tasks. While closed-source LLMs generally outperform their open-source counterparts, the progress on the latter has been rapid. This has crucial implications not only on research but also on business.
  arXiv  Detail & Related papers  (2023-11-28T17:44:51Z)
• GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition? [82.40761196684524]
  This paper centers on the evaluation of GPT-4's linguistic and visual capabilities in zero-shot visual recognition tasks. We conduct extensive experiments to evaluate GPT-4's performance across images, videos, and point clouds. Our findings show that GPT-4, enhanced with rich linguistic descriptions, significantly improves zero-shot recognition.
  arXiv  Detail & Related papers  (2023-11-27T11:29:10Z)
• DeepInception: Hypnotize Large Language Model to Be Jailbreaker [70.34096187718941]
  Large language models (LLMs) have succeeded significantly in various applications but remain susceptible to adversarial jailbreaks. We present a method to take advantage of the LLMs' personification capabilities to construct $textita virtual, nested scene. Empirically, the contents induced by our approach can achieve leading harmfulness rates with previous counterparts.
  arXiv  Detail & Related papers  (2023-11-06T15:29:30Z)
• One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Era [95.2284704286191]
  GPT-4 (a.k.a. ChatGPT plus) is one small step for generative AI (GAI) but one giant leap for artificial general intelligence (AGI) Since its official release in November 2022, ChatGPT has quickly attracted numerous users with extensive media coverage. This work is the first to survey ChatGPT with a comprehensive review of its underlying technology, applications, and challenges.
  arXiv  Detail & Related papers  (2023-04-04T06:22:09Z)
• Emerging App Issue Identification via Online Joint Sentiment-Topic Tracing [66.57888248681303]
  We propose a novel emerging issue detection approach named MERIT. Based on the AOBST model, we infer the topics negatively reflected in user reviews for one app version. Experiments on popular apps from Google Play and Apple's App Store demonstrate the effectiveness of MERIT.
  arXiv  Detail & Related papers  (2020-08-23T06:34:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.