Anti-Phishing Training Does Not Work: A Large-Scale Empirical Assessment of Multi-Modal Training Grounded in the NIST Phish Scale

• URL: http://arxiv.org/abs/2506.19899v1
• Date: Tue, 24 Jun 2025 17:57:10 GMT
• Title: Anti-Phishing Training Does Not Work: A Large-Scale Empirical Assessment of Multi-Modal Training Grounded in the NIST Phish Scale
• Authors: Andrew T. Rozema, James C. Davis,
• Abstract summary: Phishing attacks are a critical cybersecurity threat.<n>Many organizations allocate a substantial portion of their cybersecurity budgets to phishing awareness training.<n> Empirical evidence of training (in)effectiveness is essential for evidence-based cybersecurity investment.
• Score: 3.599344290726663
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Social engineering attacks using email, commonly known as phishing, are a critical cybersecurity threat. Phishing attacks often lead to operational incidents and data breaches. As a result, many organizations allocate a substantial portion of their cybersecurity budgets to phishing awareness training, driven in part by compliance requirements. However, the effectiveness of this training remains in dispute. Empirical evidence of training (in)effectiveness is essential for evidence-based cybersecurity investment and policy development. Despite recent measurement studies, two critical gaps remain in the literature: (1) we lack a validated measure of phishing lure difficulty, and (2) there are few comparisons of different types of training in real-world business settings. To fill these gaps, we conducted a large-scale study ($N = 12{,}511$) of phishing effectiveness at a US-based financial technology (``fintech'') firm. Our two-factor design compared the effect of treatments (lecture-based, interactive, and control groups) on subjects' susceptibility to phishing lures of varying complexity (using the NIST Phish Scale). The NIST Phish Scale successfully predicted behavior (click rates: 7.0\% easy to 15.0\% hard emails, p $<$ 0.001), but training showed no significant main effects on clicks (p = 0.450) or reporting (p = 0.417). Effect sizes remained below 0.01, indicating little practical value in any of the phishing trainings we deployed. Our results add to the growing evidence that phishing training is ineffective, reinforcing the importance of phishing defense-in-depth and the merit of changes to processes and technology to reduce reliance on humans, as well as rebuking the training costs necessitated by regulatory requirements.

Related papers

• Sustaining Cyber Awareness: The Long-Term Impact of Continuous Phishing Training and Emotional Triggers [1.0399614883374282]
  Phishing constitutes more than 90% of successful cyberattacks globally.<n>This study examines how continuous cybersecurity training and emotional cues affect employee susceptibility to phishing.
  arXiv  Detail & Related papers  (2025-10-31T09:15:34Z)
• Evaluating the Dynamics of Membership Privacy in Deep Learning [9.899573367308506]
  Membership inference attacks (MIAs) pose a critical threat to the privacy of training data in deep learning.<n>This paper presents a dynamic analytical framework for dissecting and quantifying privacy leakage dynamics at the individual sample level.
  arXiv  Detail & Related papers  (2025-07-31T07:09:52Z)
• AdaPhish: AI-Powered Adaptive Defense and Education Resource Against Deceptive Emails [0.0]
  AdaPhish is an AI-powered phish bowl platform that automatically anonymizes and analyzes phishing emails.<n>It achieves real-time detection and adaptation to new phishing tactics while enabling long-term tracking of phishing trends.<n>AdaPhish presents a scalable, collaborative solution for phishing detection and cybersecurity education.
  arXiv  Detail & Related papers  (2025-02-05T21:17:19Z)
• PEEK: Phishing Evolution Framework for Phishing Generation and Evolving Pattern Analysis using Large Language Models [10.455333111937598]
  Phishing remains a pervasive cyber threat, as attackers craft deceptive emails to lure victims into revealing sensitive information.<n>Deep learning has become a key component in defending against phishing attacks, but these approaches face critical limitations.<n>We propose the first Phishing Evolution FramEworK (PEEK) for augmenting phishing email datasets with respect to quality and diversity.
  arXiv  Detail & Related papers  (2024-11-18T09:03:51Z)
• Persistent Pre-Training Poisoning of LLMs [71.53046642099142]
  Our work evaluates for the first time whether language models can also be compromised during pre-training. We pre-train a series of LLMs from scratch to measure the impact of a potential poisoning adversary. Our main result is that poisoning only 0.1% of a model's pre-training dataset is sufficient for three out of four attacks to persist through post-training.
  arXiv  Detail & Related papers  (2024-10-17T16:27:13Z)
• Efficient Backdoor Defense in Multimodal Contrastive Learning: A Token-Level Unlearning Method for Mitigating Threats [52.94388672185062]
  We propose an efficient defense mechanism against backdoor threats using a concept known as machine unlearning. This entails strategically creating a small set of poisoned samples to aid the model's rapid unlearning of backdoor vulnerabilities. In the backdoor unlearning process, we present a novel token-based portion unlearning training regime.
  arXiv  Detail & Related papers  (2024-09-29T02:55:38Z)
• Content, Nudges and Incentives: A Study on the Effectiveness and Perception of Embedded Phishing Training [14.482027080866104]
  We investigate embedded phishing training in three aspects. knowledge gains from its content, nudges and reminders from the test itself, and the deterrent effect of potential consequences. Our study contributes several novel findings on the training practice.
  arXiv  Detail & Related papers  (2024-09-02T17:17:44Z)
• The Pitfalls and Promise of Conformal Inference Under Adversarial Attacks [90.52808174102157]
  In safety-critical applications such as medical imaging and autonomous driving, it is imperative to maintain both high adversarial robustness to protect against potential adversarial attacks. A notable knowledge gap remains concerning the uncertainty inherent in adversarially trained models. This study investigates the uncertainty of deep learning models by examining the performance of conformal prediction (CP) in the context of standard adversarial attacks.
  arXiv  Detail & Related papers  (2024-05-14T18:05:19Z)
• An Innovative Information Theory-based Approach to Tackle and Enhance The Transparency in Phishing Detection [23.962076093344166]
  We propose an innovative deep learning-based approach for phishing attack localization. Our method can not only predict the vulnerability of the email data but also automatically learn and figure out the most important and phishing-relevant information.
  arXiv  Detail & Related papers  (2024-02-27T00:03:07Z)
• A Study of Different Awareness Campaigns in a Company [0.0]
  Phishing is a major cyber threat to organizations that can cause financial and reputational damage. This paper examines how awareness concepts can be successfully implemented and validated.
  arXiv  Detail & Related papers  (2023-08-29T09:57:11Z)
• G$^2$uardFL: Safeguarding Federated Learning Against Backdoor Attacks through Attributed Client Graph Clustering [116.4277292854053]
  Federated Learning (FL) offers collaborative model training without data sharing. FL is vulnerable to backdoor attacks, where poisoned model weights lead to compromised system integrity. We present G$2$uardFL, a protective framework that reinterprets the identification of malicious clients as an attributed graph clustering problem.
  arXiv  Detail & Related papers  (2023-06-08T07:15:04Z)
• Adversarial training with informed data selection [53.19381941131439]
  Adrial training is the most efficient solution to defend the network against these malicious attacks. This work proposes a data selection strategy to be applied in the mini-batch training. The simulation results show that a good compromise can be obtained regarding robustness and standard accuracy.
  arXiv  Detail & Related papers  (2023-01-07T12:09:50Z)
• SoK: Human-Centered Phishing Susceptibility [4.794822439017277]
  We propose a three-stage Phishing Susceptibility Model (PSM) for explaining how humans are involved in phishing detection and prevention. This model reveals several research gaps that need to be addressed to improve users' detection performance.
  arXiv  Detail & Related papers  (2022-02-16T07:26:53Z)
• Do Gradient Inversion Attacks Make Federated Learning Unsafe? [70.0231254112197]
  Federated learning (FL) allows the collaborative training of AI models without needing to share raw data. Recent works on the inversion of deep neural networks from model gradients raised concerns about the security of FL in preventing the leakage of training data. In this work, we show that these attacks presented in the literature are impractical in real FL use-cases and provide a new baseline attack.
  arXiv  Detail & Related papers  (2022-02-14T18:33:12Z)
• On the Impact of Hard Adversarial Instances on Overfitting in Adversarial Training [70.82725772926949]
  Adversarial training is a popular method to robustify models against adversarial attacks.<n>In this work, we investigate this phenomenon from the perspective of training instances.<n>We show that the decay in generalization performance of adversarial training is a result of fitting hard adversarial instances.
  arXiv  Detail & Related papers  (2021-12-14T12:19:24Z)
• Online Adversarial Attacks [57.448101834579624]
  We formalize the online adversarial attack problem, emphasizing two key elements found in real-world use-cases. We first rigorously analyze a deterministic variant of the online threat model. We then propose algoname, a simple yet practical algorithm yielding a provably better competitive ratio for $k=2$ over the current best single threshold algorithm.
  arXiv  Detail & Related papers  (2021-03-02T20:36:04Z)
• Toward Smart Security Enhancement of Federated Learning Networks [109.20054130698797]
  In this paper, we review the vulnerabilities of federated learning networks (FLNs) and give an overview of poisoning attacks. We present a smart security enhancement framework for FLNs. Deep reinforcement learning is applied to learn the behaving patterns of the edge devices (EDs) that can provide benign training results.
  arXiv  Detail & Related papers  (2020-08-19T08:46:39Z)
• Evaluation of Federated Learning in Phishing Email Detection [24.85352882358906]
  This paper builds upon a deep neural network model, particularly RNN and BERT for phishing email detection. It analyzes the FL-entangled learning performance under various settings, including balanced and asymmetrical data distribution.
  arXiv  Detail & Related papers  (2020-07-27T03:58:00Z)
• Phishing and Spear Phishing: examples in Cyber Espionage and techniques to protect against them [91.3755431537592]
  Phishing attacks have become the most used technique in the online scams, initiating more than 91% of cyberattacks, from 2012 onwards. This study reviews how Phishing and Spear Phishing attacks are carried out by the phishers, through 5 steps which magnify the outcome.
  arXiv  Detail & Related papers  (2020-05-31T18:10:09Z)
• Overfitting in adversarially robust deep learning [86.11788847990783]
  We show that overfitting to the training set does in fact harm robust performance to a very large degree in adversarially robust training. We also show that effects such as the double descent curve do still occur in adversarially trained models, yet fail to explain the observed overfitting.
  arXiv  Detail & Related papers  (2020-02-26T15:40:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.