Johnny Still Receives Spam SMS: Assessing the Robustness of SMS Spam Detection

• URL: http://arxiv.org/abs/2609.01171v1
• Date: Tue, 01 Sep 2026 12:47:36 GMT
• Title: Johnny Still Receives Spam SMS: Assessing the Robustness of SMS Spam Detection
• Abstract summary: We evaluate the robustness of SMS anti-spam systems that end users actually rely on.<n>Our experiments reveal significant gaps in existing spam detectors' ability to identify adversarially manipulated messages.<n>To address these weaknesses, we propose a multi-model ensemble that combines adversarial training with spam classifiers diverse in architecture and tokenization.
• Score: 0.7533748789378176
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: SMS spam detection systems often achieve high accuracy in controlled environments but struggle against adversarial attacks and increasingly sophisticated spam tactics in real-world deployments. In this paper, we evaluate the robustness of SMS anti-spam systems that end users actually rely on, including commercial messaging applications, third-party anti-spam services, and publicly available open-weight models hosted on Hugging Face. We evaluate these systems under both standard and adversarial conditions, considering perceptible and state-of-the-art imperceptible attacks. We include only perturbations that we verify survive real SMS or RCS delivery, rather than lab-only artifacts. Our experiments reveal significant gaps in existing spam detectors' ability to identify adversarially manipulated messages. We further demonstrate that adversarial training alone is insufficient. Using an explicit held-out evaluation protocol, we find that robustness transfers well within a perturbation family but degrades sharply against structurally distinct, encoding-level attacks. To address these weaknesses, we propose a multi-model ensemble that combines adversarial training with spam classifiers diverse in architecture and tokenization. Our results show that this ensemble, particularly when using a minority-voting strategy, substantially improves robustness against both perceptible and imperceptible adversarial attacks while maintaining competitive classification accuracy. We also characterize the resulting precision-recall trade-off and recommend operating points for false-positive-sensitive and recall-critical deployments. These findings highlight the need for comprehensive robustness evaluations and ensemble-based defenses for building more secure SMS spam detection systems in real-world settings.

Related papers

• Automatic Red Teaming for Implicit Vulnerabilities of Text-to-Image Models [58.59008920408855]
  We propose Adversarial Probing for Implicit VulnErabilities (AdvPIE) to expose implicit vulnerabilities without requiring access to the parameters of target models.<n> AdvPIE adopts a policy agent to generate and refine implicit adversarial prompts based on the feedback from a judge agent.<n>Experiments on standard and safety-aligned T2I models show that AdvPIE1 effectively uncovers implicit vulnerabilities.
  arXiv  Detail & Related papers  (2026-09-05T13:45:09Z)
• Breaking and Defending LLM-Powered Social Media Bot Detection Systems [0.0]
  Social media bots pose a persistent threat, enabling misinformation, opinion manipulation, and the erosion of trust in online platforms.<n>To combat this, machine learning systems have been developed to detect and limit bot activity, but attackers continuously adapt through techniques such as adversarial learning and behavior imitation.<n>Recent advances in large language models (LLMs) have significantly improved bot detection by enabling deeper semantic and contextual analysis of accounts and their content.<n>We propose a robust multi-LLM defense architecture designed to preserve detection reliability under adaptive adversarial conditions.
  arXiv  Detail & Related papers  (2026-08-16T18:56:01Z)
• Who Pays the Price? Stakeholder-Centric Prompt Injection Benchmarking for Real-world Web Agents [93.19140872946842]
  Web agents driven by large language models (LLMs) are increasingly deployed in real-world environments.<n>This makes them vulnerable to prompt-injection attacks, in which seemingly benign content embeds adversarial instructions that manipulate agent behaviour.<n>We introduce textbfsysname, a benchmark to systematically categorize and attribute harm in real-world web agent systems.
  arXiv  Detail & Related papers  (2026-06-11T14:12:43Z)
• When Embedding-Based Defenses Fail: Rethinking Safety in LLM-Based Multi-Agent Systems [18.461135869777316]
  Large language model (LLM)-powered multi-agent systems (MAS) enable agents to communicate and share information, achieving strong performance on complex tasks.<n>Existing embedding-based defenses aim to detect and prune suspicious agents, but their effectiveness depends on a clear separation between the text embeddings of malicious and benign messages.<n>We analyze this failure mode theoretically and validate it empirically with three attacks, Slow Drift, Benign Wrapper, and Chaos Seeding.
  arXiv  Detail & Related papers  (2026-05-01T22:15:11Z)
• Prompt-Unknown Promotion Attacks against LLM-based Sequential Recommender Systems [51.504307822017985]
  Large language model-powered sequential recommender systems (LLM-SRSs) have recently demonstrated remarkable performance.<n>This paper investigates the item promotion attack in LLM-SRSs under a more realistic setting where both the system prompt and victim model are unknown to the attacker.
  arXiv  Detail & Related papers  (2026-04-26T10:09:26Z)
• MultiPhishGuard: An LLM-based Multi-Agent System for Phishing Email Detection [3.187381965457262]
  MultiPhishGuard is a dynamic multi-agent detection system that synergizes specialized expertise with adversarial-aware reinforcement learning.<n>Our framework employs five cooperative agents with automatically adjusted decision weights powered by a Proximal Policy Optimization reinforcement learning algorithm.<n>Experiments demonstrate that MultiPhishGuard achieves high accuracy (97.89%) with low false positive (2.73%) and false negative rates (0.20%)
  arXiv  Detail & Related papers  (2025-05-26T23:27:15Z)
• SpaLLM-Guard: Pairing SMS Spam Detection Using Open-source and Commercial LLMs [1.3198171962008958]
  We evaluate the potential of large language models (LLMs), both open-source and commercial, for SMS spam detection.<n>We compare their performance across zero-shot, few-shot, fine-tuning, and chain-of-thought prompting approaches.<n>Fine-tuning emerges as the most effective strategy, with Mixtral achieving 98.6% accuracy and a balanced false positive and false negative rate below 2%.
  arXiv  Detail & Related papers  (2025-01-09T06:00:08Z)
• Jailbreaking as a Reward Misspecification Problem [80.52431374743998]
  We propose a novel perspective that attributes this vulnerability to reward misspecification during the alignment process.<n>We introduce a metric ReGap to quantify the extent of reward misspecification and demonstrate its effectiveness.<n>We present ReMiss, a system for automated red teaming that generates adversarial prompts in a reward-misspecified space.
  arXiv  Detail & Related papers  (2024-06-20T15:12:27Z)
• SpamDam: Towards Privacy-Preserving and Adversary-Resistant SMS Spam Detection [2.0355793807035094]
  SpamDam is a SMS spam detection framework designed to overcome key challenges in detecting and understanding SMS spam. We have compiled over 76K SMS spam messages from Twitter and Weibo between 2018 and 2023, forming the largest dataset of its kind. We have rigorously tested the adversarial robustness of SMS spam detection models, introducing the novel reverse backdoor attack.
  arXiv  Detail & Related papers  (2024-04-15T06:07:10Z)
• FaultGuard: A Generative Approach to Resilient Fault Prediction in Smart Electrical Grids [53.2306792009435]
  FaultGuard is the first framework for fault type and zone classification resilient to adversarial attacks. We propose a low-complexity fault prediction model and an online adversarial training technique to enhance robustness. Our model outclasses the state-of-the-art for resilient fault prediction benchmarking, with an accuracy of up to 0.958.
  arXiv  Detail & Related papers  (2024-03-26T08:51:23Z)
• Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Information [67.78183175605761]
  Large Language Models are susceptible to adversarial prompt attacks. This vulnerability underscores a significant concern regarding the robustness and reliability of LLMs. We introduce a novel approach to detecting adversarial prompts at a token level.
  arXiv  Detail & Related papers  (2023-11-20T03:17:21Z)
• Illusory Attacks: Information-Theoretic Detectability Matters in Adversarial Attacks [76.35478518372692]
  We introduce epsilon-illusory, a novel form of adversarial attack on sequential decision-makers. Compared to existing attacks, we empirically find epsilon-illusory to be significantly harder to detect with automated methods. Our findings suggest the need for better anomaly detectors, as well as effective hardware- and system-level defenses.
  arXiv  Detail & Related papers  (2022-07-20T19:49:09Z)
• Towards Robust Speech-to-Text Adversarial Attack [78.5097679815944]
  This paper introduces a novel adversarial algorithm for attacking the state-of-the-art speech-to-text systems, namely DeepSpeech, Kaldi, and Lingvo. Our approach is based on developing an extension for the conventional distortion condition of the adversarial optimization formulation. Minimizing over this metric, which measures the discrepancies between original and adversarial samples' distributions, contributes to crafting signals very close to the subspace of legitimate speech recordings.
  arXiv  Detail & Related papers  (2021-03-15T01:51:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.