Adversarial Robustness of Phishing Email Detection: A Comparative Study of TF-IDF + Logistic Regression and Fine-Tuned DistilBERT

• URL: http://arxiv.org/abs/2607.18429v1
• Date: Mon, 20 Jul 2026 18:28:07 GMT
• Title: Adversarial Robustness of Phishing Email Detection: A Comparative Study of TF-IDF + Logistic Regression and Fine-Tuned DistilBERT
• Abstract summary: This paper reports a comparison of two phishing-detection approaches a TF-IDF + Logistic Regression baseline and a fine-tuned DistilBERT transformer trained on a unified corpus of 82,255 emails.<n>Both models exceeded 98% accuracy on clean data yet degraded sharply under adversarial testing.<n>The results show that clean-data accuracy does not predict adversarial robustness, and adversarial testing should be a standard part of phishing-detection evaluation.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Phishing emails remain one of the most persistent cybersecurity threats, and machine-learning classifiers are widely used to detect them. Most reported detection accuracies, however, are measured on clean, in-distribution test data rather than on emails deliberately altered to evade detection. This paper reports a controlled, pairwise comparison of two phishing-detection approaches a TF-IDF + Logistic Regression baseline and a fine-tuned DistilBERT transformer trained on a unified corpus of 82,255 emails drawn from six public datasets and evaluated under three conditions: normal in-distribution, synthetic phishing, and adversarial phishing. Both models exceeded 98% accuracy on clean data yet degraded sharply under adversarial testing: TF-IDF + LR fell to 64.00% (a 34.59-percentage-point drop) and DistilBERT fell to 63.64% (a 35.40-percentage-point drop) a gap of only 0.36 percentage points, equivalent to a single email in the 275-sample adversarial test set. LIME, SHAP, and attention-rollout analysis indicate the two models relied on different evidence yet showed similar vulnerability. Pairwise error analysis shows the models agreed on 54.9% of adversarial samples but each made a similar number of exclusive errors (24 and 25 respectively), indicating partly complementary rather than identical failure modes. The results show that clean-data accuracy does not predict adversarial robustness, and that adversarial testing should be a standard part of phishing-detection evaluation.

Related papers

• Not All Relations Are Equal: Relation-Balanced and Calibrated Graph Learning for Provenance-Based Intrusion Detection [66.44527094471619]
  We present RECAL, an unsupervised framework using relation-balanced masked graph learning to better capture rare interaction patterns.<n>On three DARPA E3 datasets, RECAL achieves F1 scores of 99.99%, 99.93%, and 99.99%, outperforming the best baseline on each dataset by 0.88, 0.82, and 0.42 percentage points, respectively.
  arXiv  Detail & Related papers  (2026-09-15T00:31:01Z)
• The Model Proposes, the Code Disposes: A Pre-Registered Ablation of a Verifier-and-Acceptance Stage in an LLM-Orchestrated Offensive-Security Agent [0.0]
  We evaluate whether a verifier-and-acceptance stage - a model verifier whose verdicts are enforced by deterministic code - changes what an LLM-driven offensive-security agent reports.<n>We report a 15-run exploratory pilot, a pre-registered 20-run confirmatory ablation, and a pre-registered 2 x 2 factorial study with 40 runs across two deliberately vulnerable lab targets.<n>The results support a narrow conclusion: the verifier changes what the system ships, while deterministic code supplies enforcement and auditability.
  arXiv  Detail & Related papers  (2026-09-14T17:13:05Z)
• SHIFT-M3: Pre-fusion Alignment-based Consistency Screening for Multimodal ECG Record Integrity [0.0]
  Multimodal clinical AI typically assumes that the waveform, report, metadata, and downstream predictions attached to a record belong to the same patient.<n>We study this problem as multimodal record integrity triage: given an assembled record, should its modalities be trusted to belong together?<n>We introduce SHIFT-M3, a lightweight text-based pre-fusion screen that measures alignment-based consistency between two separately produced ECG text views.
  arXiv  Detail & Related papers  (2026-09-12T10:58:02Z)
• Empirical Evaluation of Membership Inference Attacks on NLP Text Classifiers: A Baseline Study on SST-2 [0.0]
  Membership inference attacks (MIAs) try to determine whether a specific record was used to train a model.<n>This paper presents a benchmark of membership inference vulnerability for text classification on the GLUE SST-2 sentiment dataset.
  arXiv  Detail & Related papers  (2026-09-10T00:53:20Z)
• Evaluating Large Language Models for Symbolic Security Protocol Analysis [1.0499611180329804]
  This study evaluates whether Large Language Models (LLMs) can perform comparable analysis.<n>We test GPT and DeepSeek in chat and reasoning modes over three runs on 130 obfuscated AnB/AnBx protocols.
  arXiv  Detail & Related papers  (2026-07-22T20:29:59Z)
• Efficient Visual Pointing for Embodied AI:Agent-Driven Data Synthesis, Cross-Block Attention, and Iterative Correction [55.11480729304395]
  We describe our PointArena 2026 solution, which achieves 77.2% overall accuracy and ranks second on the benchmark.<n>The ap proach targets three failure modes. First, agent-driven syn thesis builds large semantic and anchor-relative candidate pools.<n>Second, a determinis tic steerable-data pipeline creates a verified 10,000-sample main set, plus reserve samples, using masks, templates, and path verification.
  arXiv  Detail & Related papers  (2026-06-29T06:39:03Z)
• Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software [1.0026496861838445]
  CWE-Trace is a framework for vulnerability detection built from 834 manually curated Linux kernel samples.<n>We evaluate eight vanilla LLMs and 15 LoRA fine-tuned variants across non-targeted detection, targeted detection, and CWE classification.
  arXiv  Detail & Related papers  (2026-06-18T17:19:26Z)
• Building an Adversarial Malware Dataset by Family and Type: Generation, Evasion, and Poisoning Evaluation [0.0]
  We present a dataset of adversarial malware samples derived from the public RawMal-TF collection of real-world malware binaries.<n>Using a suite of adversarial malware generators, we construct two sets of adversarial PE files: 44,347 family-labelled samples and 33,596 type-labelled samples.<n>Each adversarial binary is accompanied by detailed metadata, including EMBER scores and VirusTotal classifications.
  arXiv  Detail & Related papers  (2026-05-25T15:17:02Z)
• X-MAP: eXplainable Misclassification Analysis and Profiling for Spam and Phishing Detection [16.604623864453043]
  This paper presents X-MAP, an eXplainable Misclassification Analysis and Profilling framework.<n>X-MAP builds interpretable topic profiles for reliably classified spam/phishing and legitimate messages.<n>As a detector, X-MAP achieves up to 0.98 AUROC and lowers the false-rejection rate at 95% TRR to 0.089 on positive predictions.
  arXiv  Detail & Related papers  (2026-02-17T01:46:08Z)
• WISE: Web Information Satire and Fakeness Evaluation [0.9694940903078657]
  MiniLM, a lightweight model, achieves the highest accuracy (87.58%) among all models.<n>DistilBERT offers an excellent efficiency-accuracy trade-off with 86.28% accuracy and 93.90% ROC-AUC.
  arXiv  Detail & Related papers  (2025-12-30T05:44:32Z)
• Deep Reinforcement Learning for Phishing Detection with Transformer-Based Semantic Features [0.0]
  Phishing is a cybercrime in which individuals are deceived into revealing personal information.<n>This study proposes a Quantile Deep Q-Network (QR-DQN) approach that integrates RoBERTa semantic embeddings with handcrafted lexical features.<n>A diverse dataset of 105,000 URLs was curated from PhishTank, OpenPhish, Cloudflare, and other sources.
  arXiv  Detail & Related papers  (2025-12-07T17:08:12Z)
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
  We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
  arXiv  Detail & Related papers  (2025-10-02T02:14:33Z)
• BURN: Backdoor Unlearning via Adversarial Boundary Analysis [73.14147934175604]
  Backdoor unlearning aims to remove backdoor-related information while preserving the model's original functionality.<n>We propose Backdoor Unlearning via adversaRial bouNdary analysis (BURN), a novel defense framework that integrates false correlation decoupling, progressive data refinement, and model purification.
  arXiv  Detail & Related papers  (2025-07-14T17:13:06Z)
• CausalDiff: Causality-Inspired Disentanglement via Diffusion Model for Adversarial Defense [61.78357530675446]
  Humans are difficult to be cheated by subtle manipulations, since we make judgments only based on essential factors.<n>Inspired by this observation, we attempt to model label generation with essential label-causative factors and incorporate label-non-causative factors to assist data generation.<n>For an adversarial example, we aim to discriminate perturbations as non-causative factors and make predictions only based on the label-causative factors.
  arXiv  Detail & Related papers  (2024-10-30T15:06:44Z)
• Uncertainty-Calibrated Test-Time Model Adaptation without Forgetting [65.21599711087538]
  Test-time adaptation (TTA) seeks to tackle potential distribution shifts between training and test data by adapting a given model w.r.t. any test sample.<n>Prior methods perform backpropagation for each test sample, resulting in unbearable optimization costs to many applications.<n>We propose an Efficient Anti-Forgetting Test-Time Adaptation (EATA) method which develops an active sample selection criterion to identify reliable and non-redundant samples.
  arXiv  Detail & Related papers  (2024-03-18T05:49:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.