Send to which account? Evaluation of an LLM-based Scambaiting System

• URL: http://arxiv.org/abs/2509.08493v1
• Date: Wed, 10 Sep 2025 11:08:52 GMT
• Title: Send to which account? Evaluation of an LLM-based Scambaiting System
• Authors: Hossein Siadati, Haadi Jafarian, Sima Jafarikhah,
• Abstract summary: This paper presents the first large-scale, real-world evaluation of a scambaiting system powered by large language models (LLMs)<n>Over a five-month deployment, the system initiated over 2,600 engagements with actual scammers, resulting in a dataset of more than 18,700 messages.<n>It achieved an Information Disclosure Rate (IDR) of approximately 32%, successfully extracting sensitive financial information such as mule accounts.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scammers are increasingly harnessing generative AI(GenAI) technologies to produce convincing phishing content at scale, amplifying financial fraud and undermining public trust. While conventional defenses, such as detection algorithms, user training, and reactive takedown efforts remain important, they often fall short in dismantling the infrastructure scammers depend on, including mule bank accounts and cryptocurrency wallets. To bridge this gap, a proactive and emerging strategy involves using conversational honeypots to engage scammers and extract actionable threat intelligence. This paper presents the first large-scale, real-world evaluation of a scambaiting system powered by large language models (LLMs). Over a five-month deployment, the system initiated over 2,600 engagements with actual scammers, resulting in a dataset of more than 18,700 messages. It achieved an Information Disclosure Rate (IDR) of approximately 32%, successfully extracting sensitive financial information such as mule accounts. Additionally, the system maintained a Human Acceptance Rate (HAR) of around 70%, indicating strong alignment between LLM-generated responses and human operator preferences. Alongside these successes, our analysis reveals key operational challenges. In particular, the system struggled with engagement takeoff: only 48.7% of scammers responded to the initial seed message sent by defenders. These findings highlight the need for further refinement and provide actionable insights for advancing the design of automated scambaiting systems.

Related papers

• Anansi: Scalable Characterization of Message-Based Job Scams [4.132349063771989]
  Job-based smishing scams represent a rapidly growing and understudied threat within the broader landscape of online fraud.<n>Anansi is the first scalable, end-to-end measurement pipeline designed to systematically engage with, analyze, and characterize job scams in the wild.
  arXiv  Detail & Related papers  (2026-02-27T17:49:56Z)
• ReasoningBomb: A Stealthy Denial-of-Service Attack by Inducing Pathologically Long Reasoning in Large Reasoning Models [67.15960154375131]
  Large reasoning models (LRMs) extend large language models with explicit multi-step reasoning traces.<n>This capability introduces a new class of prompt-induced inference-time denial-of-service (PI-DoS) attacks that exploit the high computational cost of reasoning.<n>We present ReasoningBomb, a reinforcement-learning-based PI-DoS framework that is guided by a constant-time surrogate reward.
  arXiv  Detail & Related papers  (2026-01-29T18:53:01Z)
• SCRIPTMIND: Crime Script Inference and Cognitive Evaluation for LLM-based Social Engineering Scam Detection System [14.704193787708872]
  Social engineering scams increasingly employ personalized, multi-turn deception.<n>While Large Language Models (LLMs) show promise in identifying deception, their cognitive assistance potential remains underexplored.<n>We propose ScriptMind, an integrated framework for LLM-based scam detection that bridges automated reasoning and human cognition.
  arXiv  Detail & Related papers  (2026-01-20T04:11:00Z)
• SAFE-QAQ: End-to-End Slow-Thinking Audio-Text Fraud Detection via Reinforcement Learning [52.29460857893198]
  Existing fraud detection methods rely on transcribed text, suffering from ASR errors and missing crucial acoustic cues like vocal tone and environmental context.<n>We propose SAFE-QAQ, an end-to-end comprehensive framework for audio-based slow-thinking fraud detection.<n>Our framework introduces a dynamic risk assessment framework during live calls, enabling early detection and prevention of fraud.
  arXiv  Detail & Related papers  (2026-01-04T06:09:07Z)
• When AI Agents Collude Online: Financial Fraud Risks by Collaborative LLM Agents on Social Platforms [101.2197679948061]
  We study the risks of collective financial fraud in large-scale multi-agent systems powered by large language model (LLM) agents.<n>We present MultiAgentFraudBench, a large-scale benchmark for simulating financial fraud scenarios.
  arXiv  Detail & Related papers  (2025-11-09T16:30:44Z)
• Victim as a Service: Designing a System for Engaging with Interactive Scammers [29.43320237202651]
  We describe the motivation, design, implementation, and experience with CHATTERBOX, an LLM-based system that automates long-term engagement with online scammers.<n>We describe the techniques we have developed to attract scam attempts, the system and LLM-engineering required to convincingly engage with scammers, and the necessary capabilities required to satisfy or evade "milestones" in scammers' workflow.
  arXiv  Detail & Related papers  (2025-10-27T23:19:29Z)
• Evaluating & Reducing Deceptive Dialogue From Language Models with Multi-turn RL [64.3268313484078]
  Large Language Models (LLMs) interact with millions of people worldwide in applications such as customer support, education and healthcare.<n>Their ability to produce deceptive outputs, whether intentionally or inadvertently, poses significant safety concerns.<n>We investigate the extent to which LLMs engage in deception within dialogue, and propose the belief misalignment metric to quantify deception.
  arXiv  Detail & Related papers  (2025-10-16T05:29:36Z)
• CASE: An Agentic AI Framework for Enhancing Scam Intelligence in Digital Payments [0.24378845585726894]
  This paper presents CASE (Conversational Agent for Scam Elucidation), a novel Agentic AI framework.<n>A conversational agent is uniquely designed to proactively interview potential victims to elicit intelligence in the form of a detailed conversation.<n>By augmenting our existing features with this new intelligence, we have observed a 21% uplift in the volume of scam enforcements.
  arXiv  Detail & Related papers  (2025-08-27T14:47:33Z)
• Deep Learning Approaches for Anti-Money Laundering on Mobile Transactions: Review, Framework, and Directions [51.43521977132062]
  Money laundering is a financial crime that obscures the origin of illicit funds.<n>The proliferation of mobile payment platforms and smart IoT devices has significantly complicated anti-money laundering investigations.<n>This paper conducts a comprehensive review of deep learning solutions and the challenges associated with their use in AML.
  arXiv  Detail & Related papers  (2025-03-13T05:19:44Z)
• Human-Readable Adversarial Prompts: An Investigation into LLM Vulnerabilities Using Situational Context [45.821481786228226]
  We show that situation-driven adversarial full-prompts that leverage situational context are effective but much harder to detect.<n>We developed attacks that use movie scripts as situational contextual frameworks.<n>We enhanced the AdvPrompter framework with p-nucleus sampling to generate diverse human-readable adversarial texts.
  arXiv  Detail & Related papers  (2024-12-20T21:43:52Z)
• Exposing LLM Vulnerabilities: Adversarial Scam Detection and Performance [16.9071617169937]
  This paper investigates the vulnerabilities of Large Language Models (LLMs) when facing adversarial scam messages for the task of scam detection.<n>We created a comprehensive dataset with fine-grained labels of scam messages, including both original and adversarial scam messages.<n>Our analysis showed how adversarial examples took advantage of vulnerabilities of a LLM, leading to high misclassification rate.
  arXiv  Detail & Related papers  (2024-12-01T00:13:28Z)
• Mitigating Backdoor Threats to Large Language Models: Advancement and Challenges [46.032173498399885]
  Large Language Models (LLMs) have significantly impacted various domains, including Web search, healthcare, and software development. As these models scale, they become more vulnerable to cybersecurity risks, particularly backdoor attacks.
  arXiv  Detail & Related papers  (2024-09-30T06:31:36Z)
• Combating Phone Scams with LLM-based Detection: Where Do We Stand? [1.8979188847659796]
  This research explores the potential of large language models (LLMs) to provide detection of fraudulent phone calls. LLMs-based detectors can identify potential scams as they occur, offering immediate protection to users.
  arXiv  Detail & Related papers  (2024-09-18T02:14:30Z)
• Detecting and Understanding Vulnerabilities in Language Models via Mechanistic Interpretability [44.99833362998488]
  Large Language Models (LLMs) have shown impressive performance across a wide range of tasks. LLMs in particular are known to be vulnerable to adversarial attacks, where an imperceptible change to the input can mislead the output of the model. We propose a method, based on Mechanistic Interpretability (MI) techniques, to guide this process.
  arXiv  Detail & Related papers  (2024-07-29T09:55:34Z)
• Prompted Contextual Vectors for Spear-Phishing Detection [41.26408609344205]
  Spear-phishing attacks present a significant security challenge.<n>We propose a detection approach based on a novel document vectorization method.<n>Our method achieves a 91% F1 score in identifying LLM-generated spear-phishing emails.
  arXiv  Detail & Related papers  (2024-02-13T09:12:55Z)
• Detecting Scams Using Large Language Models [19.7220607313348]
  Large Language Models (LLMs) have gained prominence in various applications, including security. This paper explores the utility of LLMs in scam detection, a critical aspect of cybersecurity. We propose a novel use case for LLMs to identify scams, such as phishing, advance fee fraud, and romance scams.
  arXiv  Detail & Related papers  (2024-02-05T16:13:54Z)
• Automatic Scam-Baiting Using ChatGPT [0.46040036610482665]
  We report on the results of a month-long experiment comparing the effectiveness of two ChatGPT-based automatic scam-baiters to a control measure. With engagement from over 250 real email fraudsters, we find that ChatGPT-based scam-baiters show a marked increase in scammer response rate and conversation length. We discuss the implications of these results and practical considerations for wider deployment of automatic scam-baiting.
  arXiv  Detail & Related papers  (2023-09-04T13:13:35Z)
• Deep convolutional forest: a dynamic deep ensemble approach for spam detection in text [219.15486286590016]
  This paper introduces a dynamic deep ensemble model for spam detection that adjusts its complexity and extracts features automatically. As a result, the model achieved high precision, recall, f1-score and accuracy of 98.38%.
  arXiv  Detail & Related papers  (2021-10-10T17:19:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.