Zero-Shot Spam Email Classification Using Pre-trained Large Language Models

• URL: http://arxiv.org/abs/2405.15936v1
• Date: Fri, 24 May 2024 20:55:49 GMT
• Title: Zero-Shot Spam Email Classification Using Pre-trained Large Language Models
• Authors: Sergio Rojas-Galeano,
• Abstract summary: This paper investigates the application of pre-trained large language models (LLMs) for spam email classification using zero-shot prompting. We evaluate the performance of both open-source (Flan-T5) and proprietary LLMs (ChatGPT, GPT-4) on the well-known SpamAssassin dataset.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This paper investigates the application of pre-trained large language models (LLMs) for spam email classification using zero-shot prompting. We evaluate the performance of both open-source (Flan-T5) and proprietary LLMs (ChatGPT, GPT-4) on the well-known SpamAssassin dataset. Two classification approaches are explored: (1) truncated raw content from email subject and body, and (2) classification based on summaries generated by ChatGPT. Our empirical analysis, leveraging the entire dataset for evaluation without further training, reveals promising results. Flan-T5 achieves a 90% F1-score on the truncated content approach, while GPT-4 reaches a 95% F1-score using summaries. While these initial findings on a single dataset suggest the potential for classification pipelines of LLM-based subtasks (e.g., summarisation and classification), further validation on diverse datasets is necessary. The high operational costs of proprietary models, coupled with the general inference costs of LLMs, could significantly hinder real-world deployment for spam filtering.

Related papers

• SemiEvol: Semi-supervised Fine-tuning for LLM Adaptation [14.782756931646627]
  We introduce a semi-supervised fine-tuning framework named SemiEvol for LLM adaptation from a propagate-and-select manner. For knowledge propagation, SemiEvol adopts a bi-level approach, propagating knowledge from labeled data to unlabeled data through both in-weight and in-context methods. For knowledge selection, SemiEvol incorporates a collaborative learning mechanism, selecting higher-quality pseudo-response samples.
  arXiv  Detail & Related papers  (2024-10-17T16:59:46Z)
• Co-training for Low Resource Scientific Natural Language Inference [65.37685198688538]
  We propose a novel co-training method that assigns weights based on the training dynamics of the classifiers to the distantly supervised labels. By assigning importance weights instead of filtering out examples based on an arbitrary threshold on the predicted confidence, we maximize the usage of automatically labeled data. The proposed method obtains an improvement of 1.5% in Macro F1 over the distant supervision baseline, and substantial improvements over several other strong SSL baselines.
  arXiv  Detail & Related papers  (2024-06-20T18:35:47Z)
• Large Language Model-guided Document Selection [23.673690115025913]
  Large Language Model (LLM) pre-training exhausts an ever growing compute budget. Recent research has demonstrated that careful document selection enables comparable model quality with only a fraction of the FLOPs. We explore a promising direction for scalable general-domain document selection.
  arXiv  Detail & Related papers  (2024-06-07T04:52:46Z)
• Evaluating Large Language Models for Health-Related Text Classification Tasks with Public Social Media Data [3.9459077974367833]
  Large language models (LLMs) have demonstrated remarkable success in NLP tasks. We benchmarked one supervised classic machine learning model based on Support Vector Machines (SVMs), three supervised pretrained language models (PLMs) based on RoBERTa, BERTweet, and SocBERT, and two LLM based classifiers (GPT3.5 and GPT4), across 6 text classification tasks. Our comprehensive experiments demonstrate that employ-ing data augmentation using LLMs (GPT-4) with relatively small human-annotated data to train lightweight supervised classification models achieves superior results compared to training with human-annotated data
  arXiv  Detail & Related papers  (2024-03-27T22:05:10Z)
• ExtractGPT: Exploring the Potential of Large Language Models for Product Attribute Value Extraction [52.14681890859275]
  E-commerce platforms require structured product data in the form of attribute-value pairs. BERT-based extraction methods require large amounts of task-specific training data. This paper explores using large language models (LLMs) as a more training-data efficient and robust alternative.
  arXiv  Detail & Related papers  (2023-10-19T07:39:00Z)
• BooookScore: A systematic exploration of book-length summarization in the era of LLMs [53.42917858142565]
  We develop an automatic metric, BooookScore, that measures the proportion of sentences in a summary that do not contain any of the identified error types. We find that closed-source LLMs such as GPT-4 and 2 produce summaries with higher BooookScore than those generated by open-source models.
  arXiv  Detail & Related papers  (2023-10-01T20:46:44Z)
• Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting [65.00288634420812]
  Pairwise Ranking Prompting (PRP) is a technique to significantly reduce the burden on Large Language Models (LLMs) Our results are the first in the literature to achieve state-of-the-art ranking performance on standard benchmarks using moderate-sized open-sourced LLMs.
  arXiv  Detail & Related papers  (2023-06-30T11:32:25Z)
• Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Method [35.181659789684545]
  Automatic summarization generates concise summaries that contain key ideas of source documents. References from CNN/DailyMail and BBC XSum are noisy, mainly in terms of factual hallucination and information redundancy. We propose a Summary Chain-of-Thought (SumCoT) technique to elicit LLMs to generate summaries step by step. Experimental results show our method outperforms state-of-the-art fine-tuned PLMs and zero-shot LLMs by +4.33/+4.77 in ROUGE-L.
  arXiv  Detail & Related papers  (2023-05-22T18:54:35Z)
• Selective In-Context Data Augmentation for Intent Detection using Pointwise V-Information [100.03188187735624]
  We introduce a novel approach based on PLMs and pointwise V-information (PVI), a metric that can measure the usefulness of a datapoint for training a model. Our method first fine-tunes a PLM on a small seed of training data and then synthesizes new datapoints - utterances that correspond to given intents. Our method is thus able to leverage the expressive power of large language models to produce diverse training data.
  arXiv  Detail & Related papers  (2023-02-10T07:37:49Z)
• Beyond prompting: Making Pre-trained Language Models Better Zero-shot Learners by Clustering Representations [24.3378487252621]
  We show that zero-shot text classification can be improved simply by clustering texts in the embedding spaces of pre-trained language models. Our approach achieves an average of 20% absolute improvement over prompt-based zero-shot learning.
  arXiv  Detail & Related papers  (2022-10-29T16:01:51Z)
• Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function [106.69643619725652]
  We develop a training strategy that allows even a simple BiLSTM model, when trained with cross-entropy loss, to achieve competitive results. We report state-of-the-art results for text classification task on several benchmark datasets.
  arXiv  Detail & Related papers  (2020-09-08T21:55:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.