Evaluating the Efficacy of Large Language Models in Identifying Phishing Attempts

• URL: http://arxiv.org/abs/2404.15485v3
• Date: Thu, 6 Jun 2024 21:03:03 GMT
• Title: Evaluating the Efficacy of Large Language Models in Identifying Phishing Attempts
• Authors: Het Patel, Umair Rehman, Farkhund Iqbal,
• Abstract summary: Phishing, a prevalent cybercrime tactic for decades, remains a significant threat in today's digital world. This paper aims to analyze the effectiveness of 15 Large Language Models (LLMs) in detecting phishing attempts.
• Score: 2.6012482282204004
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Phishing, a prevalent cybercrime tactic for decades, remains a significant threat in today's digital world. By leveraging clever social engineering elements and modern technology, cybercrime targets many individuals, businesses, and organizations to exploit trust and security. These cyber-attackers are often disguised in many trustworthy forms to appear as legitimate sources. By cleverly using psychological elements like urgency, fear, social proof, and other manipulative strategies, phishers can lure individuals into revealing sensitive and personalized information. Building on this pervasive issue within modern technology, this paper aims to analyze the effectiveness of 15 Large Language Models (LLMs) in detecting phishing attempts, specifically focusing on a randomized set of "419 Scam" emails. The objective is to determine which LLMs can accurately detect phishing emails by analyzing a text file containing email metadata based on predefined criteria. The experiment concluded that the following models, ChatGPT 3.5, GPT-3.5-Turbo-Instruct, and ChatGPT, were the most effective in detecting phishing emails.

Related papers

• Enhancing Phishing Email Identification with Large Language Models [0.40792653193642503]
  We study the efficacy of large language models (LLMs) in detecting phishing emails. Experiments show that the LLM achieves a high accuracy rate at high precision.
  arXiv  Detail & Related papers  (2025-02-07T08:45:50Z)
• PhishIntel: Toward Practical Deployment of Reference-Based Phishing Detection [33.98293686647553]
  PhishIntel is an end-to-end phishing detection system for real-world deployment. It segmenting the detection process into two distinct tasks: a fast task that checks against local blacklists and result cache, and a slow task that conducts online blacklist verification, URL crawling, and webpage analysis. This fast-slow task system architecture ensures low response latency while retaining the robust detection capabilities of reference-based phishing detectors.
  arXiv  Detail & Related papers  (2024-12-12T08:33:39Z)
• Next-Generation Phishing: How LLM Agents Empower Cyber Attackers [10.067883724547182]
  The escalating threat of phishing emails has become increasingly sophisticated with the rise of Large Language Models (LLMs) As attackers exploit LLMs to craft more convincing and evasive phishing emails, it is crucial to assess the resilience of current phishing defenses. We conduct a comprehensive evaluation of traditional phishing detectors, such as Gmail Spam Filter, Apache SpamAssassin, and Proofpoint, as well as machine learning models like SVM, Logistic Regression, and Naive Bayes. Our results reveal notable declines in detection accuracy for rephrased emails across all detectors, highlighting critical weaknesses in current phishing defenses.
  arXiv  Detail & Related papers  (2024-11-21T06:20:29Z)
• Countering Autonomous Cyber Threats [40.00865970939829]
  Foundation Models present dual-use concerns broadly and within the cyber domain specifically. Recent research has shown the potential for these advanced models to inform or independently execute offensive cyberspace operations. This work evaluates several state-of-the-art FMs on their ability to compromise machines in an isolated network and investigates defensive mechanisms to defeat such AI-powered attacks.
  arXiv  Detail & Related papers  (2024-10-23T22:46:44Z)
• APOLLO: A GPT-based tool to detect phishing emails and generate explanations that warn users [2.3618982787621]
  Large Language Models (LLMs) offer significant promise for text processing in various domains. We present APOLLO, a tool based on OpenAI's GPT-4o to detect phishing emails and generate explanation messages. We also conducted a study with 20 participants, comparing four different explanations presented as phishing warnings.
  arXiv  Detail & Related papers  (2024-10-10T14:53:39Z)
• ChatSpamDetector: Leveraging Large Language Models for Effective Phishing Email Detection [2.3999111269325266]
  This study introduces ChatSpamDetector, a system that uses large language models (LLMs) to detect phishing emails. By converting email data into a prompt suitable for LLM analysis, the system provides a highly accurate determination of whether an email is phishing or not. We conducted an evaluation using a comprehensive phishing email dataset and compared our system to several LLMs and baseline systems.
  arXiv  Detail & Related papers  (2024-02-28T06:28:15Z)
• An Explainable Transformer-based Model for Phishing Email Detection: A Large Language Model Approach [2.8282906214258805]
  Phishing email is a serious cyber threat that tries to deceive users by sending false emails with the intention of stealing confidential information or causing financial harm. Despite extensive academic research, phishing detection remains an ongoing and formidable challenge in the cybersecurity landscape. We present an optimized, fine-tuned transformer-based DistilBERT model designed for the detection of phishing emails.
  arXiv  Detail & Related papers  (2024-02-21T15:23:21Z)
• Prompted Contextual Vectors for Spear-Phishing Detection [41.26408609344205]
  Spear-phishing attacks present a significant security challenge. We propose a detection approach based on a novel document vectorization method. Our method achieves a 91% F1 score in identifying LLM-generated spear-phishing emails.
  arXiv  Detail & Related papers  (2024-02-13T09:12:55Z)
• Detecting Backdoors in Deep Text Classifiers [43.36440869257781]
  We present the first robust defence mechanism that generalizes to several backdoor attacks against text classification models. Our technique is highly accurate at defending against state-of-the-art backdoor attacks, including data poisoning and weight poisoning.
  arXiv  Detail & Related papers  (2022-10-11T07:48:03Z)
• Deep convolutional forest: a dynamic deep ensemble approach for spam detection in text [219.15486286590016]
  This paper introduces a dynamic deep ensemble model for spam detection that adjusts its complexity and extracts features automatically. As a result, the model achieved high precision, recall, f1-score and accuracy of 98.38%.
  arXiv  Detail & Related papers  (2021-10-10T17:19:37Z)
• Phishing and Spear Phishing: examples in Cyber Espionage and techniques to protect against them [91.3755431537592]
  Phishing attacks have become the most used technique in the online scams, initiating more than 91% of cyberattacks, from 2012 onwards. This study reviews how Phishing and Spear Phishing attacks are carried out by the phishers, through 5 steps which magnify the outcome.
  arXiv  Detail & Related papers  (2020-05-31T18:10:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.