Anti-Phishing Training (Still) Does Not Work: A Large-Scale Reproduction of Phishing Training Inefficacy Grounded in the NIST Phish Scale

• URL: http://arxiv.org/abs/2506.19899v3
• Date: Mon, 13 Oct 2025 18:25:16 GMT
• Title: Anti-Phishing Training (Still) Does Not Work: A Large-Scale Reproduction of Phishing Training Inefficacy Grounded in the NIST Phish Scale
• Abstract summary: Social engineering attacks delivered via email, commonly known as phishing, represent a persistent cybersecurity threat.<n>Many organizations train employees on phishing, often mandated by compliance requirements, but the real-world effectiveness of this training remains debated.<n>We conducted a large-scale reproduction study at a US-based financial technology firm to contribute to evidence-based cybersecurity policy.
• Score: 6.661890986344375
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Social engineering attacks delivered via email, commonly known as phishing, represent a persistent cybersecurity threat leading to significant organizational incidents and data breaches. Although many organizations train employees on phishing, often mandated by compliance requirements, the real-world effectiveness of this training remains debated. To contribute to evidence-based cybersecurity policy, we conducted a large-scale reproduction study (N = 12,511) at a US-based financial technology firm. Our experimental design refined prior work by comparing training modalities in operational environments, validating NIST's standardized phishing difficulty measurement, and introducing novel organizational-level temporal resilience metrics. Echoing prior work, training interventions showed no significant main effects on click rates (p=0.450) or reporting rates (p=0.417), with negligible effect sizes. However, we found that the NIST Phish Scale predicted user behavior, with click rates increasing from 7.0% for easy lures to 15.0% for hard lures. Our organizational-level resilience result was mixed: 36-55% of campaigns achieved "inoculation" patterns where reports preceded clicks, but training did not significantly improve organizational-level temporal protection. In summary, our results confirm the ineffectiveness of current phishing training approaches while offering a refined study design for future work.

Related papers

• Veriphi: Attack-Guided Neural Network Verification with Dataset-Dependent Training Methods [0.0]
  We present a GPU-accelerated neural network verification system that combines fast adversarial attacks with formal bound certification.<n>We demonstrate that training method effectiveness is fundamentally dataset-dependent.<n>We achieve 5x verification speedup through attack-guided falsification and scale our approach to production-size models.
  arXiv  Detail & Related papers  (2026-06-16T20:02:51Z)
• TimeGuard: Channel-wise Pool Training for Backdoor Defense in Time Series Forecasting [69.01242632976161]
  Time Series Forecasting (TSF) is highly vulnerable to backdoor attacks.<n>Data entanglement induces channel-level signal dilution, rendering sample-filtering and trigger-synthesis defenses ineffective at localizing backdoors.<n>We propose a training-time backdoor defense for TSF, termed TimeGuard.
  arXiv  Detail & Related papers  (2026-05-21T11:58:46Z)
• Sustaining Cyber Awareness: The Long-Term Impact of Continuous Phishing Training and Emotional Triggers [1.0399614883374282]
  Phishing constitutes more than 90% of successful cyberattacks globally.<n>This study examines how continuous cybersecurity training and emotional cues affect employee susceptibility to phishing.
  arXiv  Detail & Related papers  (2025-10-31T09:15:34Z)
• Evaluating the Dynamics of Membership Privacy in Deep Learning [9.899573367308506]
  Membership inference attacks (MIAs) pose a critical threat to the privacy of training data in deep learning.<n>This paper presents a dynamic analytical framework for dissecting and quantifying privacy leakage dynamics at the individual sample level.
  arXiv  Detail & Related papers  (2025-07-31T07:09:52Z)
• AdaPhish: AI-Powered Adaptive Defense and Education Resource Against Deceptive Emails [0.0]
  AdaPhish is an AI-powered phish bowl platform that automatically anonymizes and analyzes phishing emails.<n>It achieves real-time detection and adaptation to new phishing tactics while enabling long-term tracking of phishing trends.<n>AdaPhish presents a scalable, collaborative solution for phishing detection and cybersecurity education.
  arXiv  Detail & Related papers  (2025-02-05T21:17:19Z)
• PEEK: Phishing Evolution Framework for Phishing Generation and Evolving Pattern Analysis using Large Language Models [10.455333111937598]
  Phishing remains a pervasive cyber threat, as attackers craft deceptive emails to lure victims into revealing sensitive information.<n>Deep learning has become a key component in defending against phishing attacks, but these approaches face critical limitations.<n>We propose the first Phishing Evolution FramEworK (PEEK) for augmenting phishing email datasets with respect to quality and diversity.
  arXiv  Detail & Related papers  (2024-11-18T09:03:51Z)
• Persistent Pre-Training Poisoning of LLMs [71.53046642099142]
  Our work evaluates for the first time whether language models can also be compromised during pre-training. We pre-train a series of LLMs from scratch to measure the impact of a potential poisoning adversary. Our main result is that poisoning only 0.1% of a model's pre-training dataset is sufficient for three out of four attacks to persist through post-training.
  arXiv  Detail & Related papers  (2024-10-17T16:27:13Z)
• Efficient Backdoor Defense in Multimodal Contrastive Learning: A Token-Level Unlearning Method for Mitigating Threats [52.94388672185062]
  We propose an efficient defense mechanism against backdoor threats using a concept known as machine unlearning. This entails strategically creating a small set of poisoned samples to aid the model's rapid unlearning of backdoor vulnerabilities. In the backdoor unlearning process, we present a novel token-based portion unlearning training regime.
  arXiv  Detail & Related papers  (2024-09-29T02:55:38Z)
• Content, Nudges and Incentives: A Study on the Effectiveness and Perception of Embedded Phishing Training [14.482027080866104]
  We investigate embedded phishing training in three aspects. knowledge gains from its content, nudges and reminders from the test itself, and the deterrent effect of potential consequences. Our study contributes several novel findings on the training practice.
  arXiv  Detail & Related papers  (2024-09-02T17:17:44Z)
• The Pitfalls and Promise of Conformal Inference Under Adversarial Attacks [90.52808174102157]
  In safety-critical applications such as medical imaging and autonomous driving, it is imperative to maintain both high adversarial robustness to protect against potential adversarial attacks. A notable knowledge gap remains concerning the uncertainty inherent in adversarially trained models. This study investigates the uncertainty of deep learning models by examining the performance of conformal prediction (CP) in the context of standard adversarial attacks.
  arXiv  Detail & Related papers  (2024-05-14T18:05:19Z)
• An Innovative Information Theory-based Approach to Tackle and Enhance The Transparency in Phishing Detection [23.962076093344166]
  We propose an innovative deep learning-based approach for phishing attack localization. Our method can not only predict the vulnerability of the email data but also automatically learn and figure out the most important and phishing-relevant information.
  arXiv  Detail & Related papers  (2024-02-27T00:03:07Z)
• A Study of Different Awareness Campaigns in a Company [0.0]
  Phishing is a major cyber threat to organizations that can cause financial and reputational damage. This paper examines how awareness concepts can be successfully implemented and validated.
  arXiv  Detail & Related papers  (2023-08-29T09:57:11Z)
• G$^2$uardFL: Safeguarding Federated Learning Against Backdoor Attacks through Attributed Client Graph Clustering [116.4277292854053]
  Federated Learning (FL) offers collaborative model training without data sharing. FL is vulnerable to backdoor attacks, where poisoned model weights lead to compromised system integrity. We present G$2$uardFL, a protective framework that reinterprets the identification of malicious clients as an attributed graph clustering problem.
  arXiv  Detail & Related papers  (2023-06-08T07:15:04Z)
• Adversarial training with informed data selection [53.19381941131439]
  Adrial training is the most efficient solution to defend the network against these malicious attacks. This work proposes a data selection strategy to be applied in the mini-batch training. The simulation results show that a good compromise can be obtained regarding robustness and standard accuracy.
  arXiv  Detail & Related papers  (2023-01-07T12:09:50Z)
• SoK: Human-Centered Phishing Susceptibility [4.794822439017277]
  We propose a three-stage Phishing Susceptibility Model (PSM) for explaining how humans are involved in phishing detection and prevention. This model reveals several research gaps that need to be addressed to improve users' detection performance.
  arXiv  Detail & Related papers  (2022-02-16T07:26:53Z)
• Do Gradient Inversion Attacks Make Federated Learning Unsafe? [70.0231254112197]
  Federated learning (FL) allows the collaborative training of AI models without needing to share raw data. Recent works on the inversion of deep neural networks from model gradients raised concerns about the security of FL in preventing the leakage of training data. In this work, we show that these attacks presented in the literature are impractical in real FL use-cases and provide a new baseline attack.
  arXiv  Detail & Related papers  (2022-02-14T18:33:12Z)
• On the Impact of Hard Adversarial Instances on Overfitting in Adversarial Training [70.82725772926949]
  Adversarial training is a popular method to robustify models against adversarial attacks.<n>In this work, we investigate this phenomenon from the perspective of training instances.<n>We show that the decay in generalization performance of adversarial training is a result of fitting hard adversarial instances.
  arXiv  Detail & Related papers  (2021-12-14T12:19:24Z)
• Online Adversarial Attacks [57.448101834579624]
  We formalize the online adversarial attack problem, emphasizing two key elements found in real-world use-cases. We first rigorously analyze a deterministic variant of the online threat model. We then propose algoname, a simple yet practical algorithm yielding a provably better competitive ratio for $k=2$ over the current best single threshold algorithm.
  arXiv  Detail & Related papers  (2021-03-02T20:36:04Z)
• Toward Smart Security Enhancement of Federated Learning Networks [109.20054130698797]
  In this paper, we review the vulnerabilities of federated learning networks (FLNs) and give an overview of poisoning attacks. We present a smart security enhancement framework for FLNs. Deep reinforcement learning is applied to learn the behaving patterns of the edge devices (EDs) that can provide benign training results.
  arXiv  Detail & Related papers  (2020-08-19T08:46:39Z)
• Evaluation of Federated Learning in Phishing Email Detection [24.85352882358906]
  This paper builds upon a deep neural network model, particularly RNN and BERT for phishing email detection. It analyzes the FL-entangled learning performance under various settings, including balanced and asymmetrical data distribution.
  arXiv  Detail & Related papers  (2020-07-27T03:58:00Z)
• Phishing and Spear Phishing: examples in Cyber Espionage and techniques to protect against them [91.3755431537592]
  Phishing attacks have become the most used technique in the online scams, initiating more than 91% of cyberattacks, from 2012 onwards. This study reviews how Phishing and Spear Phishing attacks are carried out by the phishers, through 5 steps which magnify the outcome.
  arXiv  Detail & Related papers  (2020-05-31T18:10:09Z)
• Overfitting in adversarially robust deep learning [86.11788847990783]
  We show that overfitting to the training set does in fact harm robust performance to a very large degree in adversarially robust training. We also show that effects such as the double descent curve do still occur in adversarially trained models, yet fail to explain the observed overfitting.
  arXiv  Detail & Related papers  (2020-02-26T15:40:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.