STF: Sentence Transformer Fine-Tuning For Topic Categorization With Limited Data

• URL: http://arxiv.org/abs/2407.03253v1
• Date: Wed, 3 Jul 2024 16:34:56 GMT
• Title: STF: Sentence Transformer Fine-Tuning For Topic Categorization With Limited Data
• Authors: Kheir Eddine Daouadi, Yaakoub Boualleg, Oussama Guehairia,
• Abstract summary: Sentence Transformers Fine-tuning (STF) is a topic detection system that leverages pretrained Sentence Transformers models and fine-tuning to classify topics from tweets accurately. Our main contribution is the achievement of promising results in tweet topic classification by applying pretrained sentence transformers language models.
• Score: 0.27309692684728604
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Nowadays, topic classification from tweets attracts considerable research attention. Different classification systems have been suggested thanks to these research efforts. Nevertheless, they face major challenges owing to low performance metrics due to the limited amount of labeled data. We propose Sentence Transformers Fine-tuning (STF), a topic detection system that leverages pretrained Sentence Transformers models and fine-tuning to classify topics from tweets accurately. Moreover, extensive parameter sensitivity analyses were conducted to finetune STF parameters for our topic classification task to achieve the best performance results. Experiments on two benchmark datasets demonstrated that (1) the proposed STF can be effectively used for classifying tweet topics and outperforms the latest state-of-the-art approaches, and (2) the proposed STF does not require a huge amount of labeled tweets to achieve good accuracy, which is a limitation of many state-of-the-art approaches. Our main contribution is the achievement of promising results in tweet topic classification by applying pretrained sentence transformers language models.

Related papers

• Depression detection in social media posts using transformer-based models and auxiliary features [6.390468088226495]
  Detection of depression in social media posts is crucial due to the increasing prevalence of mental health issues. Traditional machine learning algorithms often fail to capture intricate textual patterns, limiting their effectiveness in identifying depression. This research proposes a neural network architecture leveraging transformer-based models combined with metadata and linguistic markers.
  arXiv  Detail & Related papers  (2024-09-30T07:53:39Z)
• MoE-FFD: Mixture of Experts for Generalized and Parameter-Efficient Face Forgery Detection [54.545054873239295]
  Deepfakes have recently raised significant trust issues and security concerns among the public. ViT-based methods take advantage of the expressivity of transformers, achieving superior detection performance. This work introduces Mixture-of-Experts modules for Face Forgery Detection (MoE-FFD), a generalized yet parameter-efficient ViT-based approach.
  arXiv  Detail & Related papers  (2024-04-12T13:02:08Z)
• MisRoB{\AE}RTa: Transformers versus Misinformation [0.6091702876917281]
  We propose a novel transformer-based deep neural ensemble architecture for misinformation detection. MisRoBAERTa takes advantage of two transformers (BART & RoBERTa) to improve the classification performance. For training and testing, we used a large real-world news articles dataset labeled with 10 classes.
  arXiv  Detail & Related papers  (2023-04-16T12:14:38Z)
• Strong Baselines for Parameter Efficient Few-Shot Fine-tuning [50.83426196335385]
  Few-shot classification (FSC) entails learning novel classes given only a few examples per class after a pre-training (or meta-training) phase. Recent works have shown that simply fine-tuning a pre-trained Vision Transformer (ViT) on new test classes is a strong approach for FSC. Fine-tuning ViTs, however, is expensive in time, compute and storage. This has motivated the design of parameter efficient fine-tuning (PEFT) methods which fine-tune only a fraction of the Transformer's parameters.
  arXiv  Detail & Related papers  (2023-04-04T16:14:39Z)
• Transformers are Short Text Classifiers: A Study of Inductive Short Text Classifiers on Benchmarks and Real-world Datasets [2.9443230571766854]
  Short text classification is a crucial and challenging aspect of Natural Language Processing. In recent short text research, State of the Art (SOTA) methods for traditional text classification have been unexploited. Our experiments unambiguously demonstrate that Transformers achieve SOTA accuracy on short text classification tasks.
  arXiv  Detail & Related papers  (2022-11-30T10:25:24Z)
• Paragraph-based Transformer Pre-training for Multi-Sentence Inference [99.59693674455582]
  We show that popular pre-trained transformers perform poorly when used for fine-tuning on multi-candidate inference tasks. We then propose a new pre-training objective that models the paragraph-level semantics across multiple input sentences.
  arXiv  Detail & Related papers  (2022-05-02T21:41:14Z)
• Guiding Generative Language Models for Data Augmentation in Few-Shot Text Classification [59.698811329287174]
  We leverage GPT-2 for generating artificial training instances in order to improve classification performance. Our results show that fine-tuning GPT-2 in a handful of label instances leads to consistent classification improvements.
  arXiv  Detail & Related papers  (2021-11-17T12:10:03Z)
• Task Adaptive Pretraining of Transformers for Hostility Detection [11.306581296760864]
  We study two problems, namely, (a) Coarse binary classification of Hindi Tweets into Hostile or Not, and (b) Fine-grained multi-label classification of Tweets into four categories: hate, fake, offensive, and defamation. Our system ranked first in the 'Hostile Post Detection in Hindi' shared task with an F1 score of 97.16% for coarse-grained detection and a weighted F1 score of 62.96% for fine-grained multi-label classification on the provided blind test corpora.
  arXiv  Detail & Related papers  (2021-01-09T15:45:26Z)
• Pretrained Transformers for Text Ranking: BERT and Beyond [53.83210899683987]
  This survey provides an overview of text ranking with neural network architectures known as transformers. The combination of transformers and self-supervised pretraining has been responsible for a paradigm shift in natural language processing.
  arXiv  Detail & Related papers  (2020-10-13T15:20:32Z)
• Rank over Class: The Untapped Potential of Ranking in Natural Language Processing [8.637110868126546]
  We argue that many tasks which are currently addressed using classification are in fact being shoehorned into a classification mould. We propose a novel end-to-end ranking approach consisting of a Transformer network responsible for producing representations for a pair of text sequences. In an experiment on a heavily-skewed sentiment analysis dataset, converting ranking results to classification labels yields an approximately 22% improvement over state-of-the-art text classification.
  arXiv  Detail & Related papers  (2020-09-10T22:18:57Z)
• The Cascade Transformer: an Application for Efficient Answer Sentence Selection [116.09532365093659]
  We introduce the Cascade Transformer, a technique to adapt transformer-based models into a cascade of rankers. When compared to a state-of-the-art transformer model, our approach reduces computation by 37% with almost no impact on accuracy.
  arXiv  Detail & Related papers  (2020-05-05T23:32:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.