PsyScam: A Benchmark for Psychological Techniques in Real-World Scams

• URL: http://arxiv.org/abs/2505.15017v2
• Date: Mon, 22 Sep 2025 15:55:41 GMT
• Title: PsyScam: A Benchmark for Psychological Techniques in Real-World Scams
• Authors: Shang Ma, Tianyi Ma, Jiahao Liu, Wei Song, Zhenkai Liang, Xusheng Xiao, Yanfang Ye,
• Abstract summary: PsyScam is a benchmark designed to systematically capture the psychological techniques employed in real-world scam reports.<n>We show that PsyScam presents significant challenges to existing models in both detecting and generating scam content based on the PTs used by real-world scammers.
• Score: 38.57446009573742
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Over the years, online scams have grown dramatically, with nearly 50% of global consumers encountering scam attempts each week. These scams cause not only significant financial losses to individuals and businesses, but also lasting psychological trauma, largely due to scammers' strategic employment of psychological techniques (PTs) to manipulate victims. Meanwhile, scammers continually evolve their tactics by leveraging advances in Large Language Models (LLMs) to generate diverse scam variants that easily bypass existing defenses. To address this pressing problem, we introduce PsyScam, a benchmark designed to systematically capture the PTs employed in real-world scam reports, and investigate how LLMs can be utilized to generate variants of scams based on the PTs and the contexts provided by these scams. Specifically, we collect a wide range of scam reports and ground its annotations of employed PTs in well-established cognitive and psychological theories. We further demonstrate LLMs' capabilities in generating through two downstream tasks: scam completion, and scam augmentation. Experimental results show that PsyScam presents significant challenges to existing models in both detecting and generating scam content based on the PTs used by real-world scammers. Our code and dataset are available.

Related papers

• FraudShield: Knowledge Graph Empowered Defense for LLMs against Fraud Attacks [42.60338754083519]
  Large language models (LLMs) have been widely integrated into critical automated processes.<n>They are susceptible to manipulation by fraudulent information, which can lead to harmful outcomes.<n>We introduce FraudShield, a framework designed to protect LLMs from fraudulent content by leveraging a comprehensive analysis of fraud tactics.
  arXiv  Detail & Related papers  (2026-01-30T02:48:52Z)
• When AI Agents Collude Online: Financial Fraud Risks by Collaborative LLM Agents on Social Platforms [101.2197679948061]
  We study the risks of collective financial fraud in large-scale multi-agent systems powered by large language model (LLM) agents.<n>We present MultiAgentFraudBench, a large-scale benchmark for simulating financial fraud scenarios.
  arXiv  Detail & Related papers  (2025-11-09T16:30:44Z)
• Victim as a Service: Designing a System for Engaging with Interactive Scammers [29.43320237202651]
  We describe the motivation, design, implementation, and experience with CHATTERBOX, an LLM-based system that automates long-term engagement with online scammers.<n>We describe the techniques we have developed to attract scam attempts, the system and LLM-engineering required to convincingly engage with scammers, and the necessary capabilities required to satisfy or evade "milestones" in scammers' workflow.
  arXiv  Detail & Related papers  (2025-10-27T23:19:29Z)
• Strategic Dishonesty Can Undermine AI Safety Evaluations of Frontier LLMs [95.06033929366203]
  Large language models (LLM) developers aim for their models to be honest, helpful, and harmless.<n>We show that frontier LLMs can develop a preference for dishonesty as a new strategy, even when other options are available.<n>We find no apparent cause for the propensity to deceive, but show that more capable models are better at executing this strategy.
  arXiv  Detail & Related papers  (2025-09-22T17:30:56Z)
• One Token to Fool LLM-as-a-Judge [52.45386385722788]
  Large language models (LLMs) are increasingly trusted as automated judges, assisting evaluation and providing reward signals for training other models.<n>We uncover a critical vulnerability even in this reference-based paradigm: generative reward models are systematically susceptible to reward hacking.
  arXiv  Detail & Related papers  (2025-07-11T17:55:22Z)
• LLMs unlock new paths to monetizing exploits [85.60539289753564]
  Large language models (LLMs) will soon alter the economics of cyberattacks.<n>LLMs enable adversaries to launch tailored attacks on a user-by-user basis.
  arXiv  Detail & Related papers  (2025-05-16T17:05:25Z)
• AI persuading AI vs AI persuading Humans: LLMs' Differential Effectiveness in Promoting Pro-Environmental Behavior [70.24245082578167]
  Pro-environmental behavior (PEB) is vital to combat climate change, yet turning awareness into intention and action remains elusive.<n>We explore large language models (LLMs) as tools to promote PEB, comparing their impact across 3,200 participants.<n>Results reveal a "synthetic persuasion paradox": synthetic and simulated agents significantly affect their post-intervention PEB stance, while human responses barely shift.
  arXiv  Detail & Related papers  (2025-03-03T21:40:55Z)
• "It Warned Me Just at the Right Moment": Exploring LLM-based Real-time Detection of Phone Scams [21.992539308179126]
  We propose a framework for modeling scam calls and introduce an LLM-based real-time detection approach.<n>We evaluate the method's performance and analyze key factors influencing its effectiveness.
  arXiv  Detail & Related papers  (2025-02-06T10:57:05Z)
• Exposing LLM Vulnerabilities: Adversarial Scam Detection and Performance [16.9071617169937]
  This paper investigates the vulnerabilities of Large Language Models (LLMs) when facing adversarial scam messages for the task of scam detection.<n>We created a comprehensive dataset with fine-grained labels of scam messages, including both original and adversarial scam messages.<n>Our analysis showed how adversarial examples took advantage of vulnerabilities of a LLM, leading to high misclassification rate.
  arXiv  Detail & Related papers  (2024-12-01T00:13:28Z)
• Deepfake detection in videos with multiple faces using geometric-fakeness features [79.16635054977068]
  Deepfakes of victims or public figures can be used by fraudsters for blackmailing, extorsion and financial fraud. In our research we propose to use geometric-fakeness features (GFF) that characterize a dynamic degree of a face presence in a video. We employ our approach to analyze videos with multiple faces that are simultaneously present in a video.
  arXiv  Detail & Related papers  (2024-10-10T13:10:34Z)
• Combating Phone Scams with LLM-based Detection: Where Do We Stand? [1.8979188847659796]
  This research explores the potential of large language models (LLMs) to provide detection of fraudulent phone calls. LLMs-based detectors can identify potential scams as they occur, offering immediate protection to users.
  arXiv  Detail & Related papers  (2024-09-18T02:14:30Z)
• Automatic Scam-Baiting Using ChatGPT [0.46040036610482665]
  We report on the results of a month-long experiment comparing the effectiveness of two ChatGPT-based automatic scam-baiters to a control measure. With engagement from over 250 real email fraudsters, we find that ChatGPT-based scam-baiters show a marked increase in scammer response rate and conversation length. We discuss the implications of these results and practical considerations for wider deployment of automatic scam-baiting.
  arXiv  Detail & Related papers  (2023-09-04T13:13:35Z)
• Tainted Love: A Systematic Review of Online Romance Fraud [68.8204255655161]
  Romance fraud involves cybercriminals engineering a romantic relationship on online dating platforms. We characterise the literary landscape on romance fraud, advancing the understanding of researchers and practitioners. Three main contributions were identified: profiles of romance scams, countermeasures for mitigating romance scams, and factors that predispose an individual to become a scammer or a victim.
  arXiv  Detail & Related papers  (2023-02-28T20:34:07Z)
• Combat AI With AI: Counteract Machine-Generated Fake Restaurant Reviews on Social Media [77.34726150561087]
  We propose to leverage the high-quality elite Yelp reviews to generate fake reviews from the OpenAI GPT review creator. We apply the model to predict non-elite reviews and identify the patterns across several dimensions. We show that social media platforms are continuously challenged by machine-generated fake reviews.
  arXiv  Detail & Related papers  (2023-02-10T19:40:10Z)
• Recent trends in Social Engineering Scams and Case study of Gift Card Scam [4.345672405192058]
  Social engineering scams (SES) has been existed since the adoption of the telecommunications by humankind. Recent trends of various social engineering scams targeting the innocent people all over the world. Case study of real-time gift card scam targeting various enterprise organization customers.
  arXiv  Detail & Related papers  (2021-10-13T04:17:02Z)
• Evaluating the Robustness of Semantic Segmentation for Autonomous Driving against Real-World Adversarial Patch Attacks [62.87459235819762]
  In a real-world scenario like autonomous driving, more attention should be devoted to real-world adversarial examples (RWAEs) This paper presents an in-depth evaluation of the robustness of popular SS models by testing the effects of both digital and real-world adversarial patches.
  arXiv  Detail & Related papers  (2021-08-13T11:49:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.