Distinguishing Scams and Fraud with Ensemble Learning

• URL: http://arxiv.org/abs/2412.08680v1
• Date: Wed, 11 Dec 2024 18:07:18 GMT
• Title: Distinguishing Scams and Fraud with Ensemble Learning
• Authors: Isha Chadalavada, Tianhui Huang, Jessica Staddon,
• Abstract summary: The Consumer Financial Protection Bureau's complaints database is a rich data source for evaluating LLM performance on user scam queries. We developed an ensemble approach to distinguishing scam and fraud CFPB complaints.
• Score: 0.8192907805418583
• License:
• Abstract: Users increasingly query LLM-enabled web chatbots for help with scam defense. The Consumer Financial Protection Bureau's complaints database is a rich data source for evaluating LLM performance on user scam queries, but currently the corpus does not distinguish between scam and non-scam fraud. We developed an LLM ensemble approach to distinguishing scam and fraud CFPB complaints and describe initial findings regarding the strengths and weaknesses of LLMs in the scam defense context.

Related papers

• Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducements [23.99500412996251]
  Fraud-R1 comprises 8,564 fraud cases sourced from phishing scams, fake job postings, social media, and news, categorized into 5 major fraud types. Unlike previous benchmarks, Fraud-R1 introduces a multi-round evaluation pipeline to assess LLMs' resistance to fraud at different stages. Our evaluation reveals the significant challenges in defending against fraud and phishing inducement, especially in role-play settings and fake job postings.
  arXiv  Detail & Related papers  (2025-02-18T14:47:02Z)
• ScamFerret: Detecting Scam Websites Autonomously with Large Language Models [2.6217304977339473]
  ScamFerret is an innovative agent system employing a large language model (LLM) to autonomously collect and analyze data from a given URL to determine whether it is a scam. Our evaluation demonstrated that ScamFerret achieves 0.972 accuracy in classifying four scam types in English and 0.993 accuracy in classifying online shopping websites across three different languages.
  arXiv  Detail & Related papers  (2025-02-14T12:16:38Z)
• Token Highlighter: Inspecting and Mitigating Jailbreak Prompts for Large Language Models [61.916827858666906]
  Large Language Models (LLMs) are increasingly being integrated into services such as ChatGPT to provide responses to user queries. This paper proposes a method called Token Highlighter to inspect and mitigate the potential jailbreak threats in the user query.
  arXiv  Detail & Related papers  (2024-12-24T05:10:02Z)
• Exposing LLM Vulnerabilities: Adversarial Scam Detection and Performance [16.9071617169937]
  This paper investigates the vulnerabilities of Large Language Models (LLMs) when facing adversarial scam messages for the task of scam detection. We created a comprehensive dataset with fine-grained labels of scam messages, including both original and adversarial scam messages. Our analysis showed how adversarial examples took advantage of vulnerabilities of a LLM, leading to high misclassification rate.
  arXiv  Detail & Related papers  (2024-12-01T00:13:28Z)
• Combating Phone Scams with LLM-based Detection: Where Do We Stand? [1.8979188847659796]
  This research explores the potential of large language models (LLMs) to provide detection of fraudulent phone calls. LLMs-based detectors can identify potential scams as they occur, offering immediate protection to users.
  arXiv  Detail & Related papers  (2024-09-18T02:14:30Z)
• From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning [91.79567270986901]
  Large Language Models (LLMs) tend to prioritize adherence to user prompts over providing veracious responses. Recent works propose to employ supervised fine-tuning (SFT) to mitigate the sycophancy issue. We propose a novel supervised pinpoint tuning (SPT), where the region-of-interest modules are tuned for a given objective.
  arXiv  Detail & Related papers  (2024-09-03T07:01:37Z)
• CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models [60.59638232596912]
  We introduce CLAMBER, a benchmark for evaluating large language models (LLMs) Building upon the taxonomy, we construct 12K high-quality data to assess the strengths, weaknesses, and potential risks of various off-the-shelf LLMs. Our findings indicate the limited practical utility of current LLMs in identifying and clarifying ambiguous user queries.
  arXiv  Detail & Related papers  (2024-05-20T14:34:01Z)
• Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMs [60.40396361115776]
  This paper introduces a novel collaborative approach, namely SlimPLM, that detects missing knowledge in large language models (LLMs) with a slim proxy model. We employ a proxy model which has far fewer parameters, and take its answers as answers. Heuristic answers are then utilized to predict the knowledge required to answer the user question, as well as the known and unknown knowledge within the LLM.
  arXiv  Detail & Related papers  (2024-02-19T11:11:08Z)
• Detecting Scams Using Large Language Models [19.7220607313348]
  Large Language Models (LLMs) have gained prominence in various applications, including security. This paper explores the utility of LLMs in scam detection, a critical aspect of cybersecurity. We propose a novel use case for LLMs to identify scams, such as phishing, advance fee fraud, and romance scams.
  arXiv  Detail & Related papers  (2024-02-05T16:13:54Z)
• SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks [99.23352758320945]
  We propose SmoothLLM, the first algorithm designed to mitigate jailbreaking attacks on large language models (LLMs) Based on our finding that adversarially-generated prompts are brittle to character-level changes, our defense first randomly perturbs multiple copies of a given input prompt, and then aggregates the corresponding predictions to detect adversarial inputs.
  arXiv  Detail & Related papers  (2023-10-05T17:01:53Z)
• Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback [127.75419038610455]
  Large language models (LLMs) are able to generate human-like, fluent responses for many downstream tasks. This paper proposes a LLM-Augmenter system, which augments a black-box LLM with a set of plug-and-play modules.
  arXiv  Detail & Related papers  (2023-02-24T18:48:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.