Adaptation of domain-specific transformer models with text oversampling for sentiment analysis of social media posts on Covid-19 vaccines

• URL: http://arxiv.org/abs/2209.10966v1
• Date: Thu, 22 Sep 2022 12:36:40 GMT
• Title: Adaptation of domain-specific transformer models with text oversampling for sentiment analysis of social media posts on Covid-19 vaccines
• Authors: Anmol Bansal, Arjun Choudhry, Anubhav Sharma, Seba Susan
• Abstract summary: Covid-19 has spread across the world and several vaccines have been developed to counter its surge. To identify the correct sentiments associated with the vaccines from social media posts, we fine-tune various state-of-the-art pre-trained transformer models on tweets associated with Covid-19 vaccines.
• Score: 8.115075181267105
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Covid-19 has spread across the world and several vaccines have been developed to counter its surge. To identify the correct sentiments associated with the vaccines from social media posts, we fine-tune various state-of-the-art pre-trained transformer models on tweets associated with Covid-19 vaccines. Specifically, we use the recently introduced state-of-the-art pre-trained transformer models RoBERTa, XLNet and BERT, and the domain-specific transformer models CT-BERT and BERTweet that are pre-trained on Covid-19 tweets. We further explore the option of text augmentation by oversampling using Language Model based Oversampling Technique (LMOTE) to improve the accuracies of these models, specifically, for small sample datasets where there is an imbalanced class distribution among the positive, negative and neutral sentiment classes. Our results summarize our findings on the suitability of text oversampling for imbalanced small sample datasets that are used to fine-tune state-of-the-art pre-trained transformer models, and the utility of domain-specific transformer models for the classification task.

Related papers

• Efficient and Accurate Pneumonia Detection Using a Novel Multi-Scale Transformer Approach [1.2233362977312945]
  We propose a novel approach combining deep learning and transformer-based attention mechanisms to enhance pneumonia detection from chest X-rays. Our method begins with lung segmentation using a TransUNet model that integrates our specialized transformer module. Our approach achieves high accuracy rates of 92.79% on the Kermany dataset and 95.11% on the Cohen dataset.
  arXiv  Detail & Related papers  (2024-08-08T08:06:42Z)
• RAT: Retrieval-Augmented Transformer for Click-Through Rate Prediction [68.34355552090103]
  This paper develops a Retrieval-Augmented Transformer (RAT), aiming to acquire fine-grained feature interactions within and across samples. We then build Transformer layers with cascaded attention to capture both intra- and cross-sample feature interactions. Experiments on real-world datasets substantiate the effectiveness of RAT and suggest its advantage in long-tail scenarios.
  arXiv  Detail & Related papers  (2024-04-02T19:14:23Z)
• Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning [73.75282761503581]
  We propose DiffTPT, which leverages pre-trained diffusion models to generate diverse and informative new data. Our experiments on test datasets with distribution shifts and unseen categories demonstrate that DiffTPT improves the zero-shot accuracy by an average of 5.13%.
  arXiv  Detail & Related papers  (2023-08-11T09:36:31Z)
• MisRoB{\AE}RTa: Transformers versus Misinformation [0.6091702876917281]
  We propose a novel transformer-based deep neural ensemble architecture for misinformation detection. MisRoBAERTa takes advantage of two transformers (BART & RoBERTa) to improve the classification performance. For training and testing, we used a large real-world news articles dataset labeled with 10 classes.
  arXiv  Detail & Related papers  (2023-04-16T12:14:38Z)
• Transformer-based approaches to Sentiment Detection [55.41644538483948]
  We examined the performance of four different types of state-of-the-art transformer models for text classification. The RoBERTa transformer model performs best on the test dataset with a score of 82.6% and is highly recommended for quality predictions.
  arXiv  Detail & Related papers  (2023-03-13T17:12:03Z)
• Dense Feature Memory Augmented Transformers for COVID-19 Vaccination Search Classification [60.49594822215981]
  This paper presents a classification model for detecting COVID-19 vaccination related search queries. We propose a novel approach of considering dense features as memory tokens that the model can attend to. We show that this new modeling approach enables a significant improvement to the Vaccine Search Insights (VSI) task.
  arXiv  Detail & Related papers  (2022-12-16T13:57:41Z)
• Forecasting COVID-19 Caseloads Using Unsupervised Embedding Clusters of Social Media Posts [14.201816626446885]
  We present a novel approach incorporating transformer-based language models into infectious disease modelling. We benchmark these clustered embedding features against features extracted from other high-quality datasets. In a threshold-classification task, we show that they outperform all other feature types at predicting upward trend signals.
  arXiv  Detail & Related papers  (2022-05-20T18:59:04Z)
• MiDAS: Multi-integrated Domain Adaptive Supervision for Fake News Detection [3.210653757360955]
  We propose MiDAS, a multi-domain adaptative approach for fake news detection. MiDAS ranks relevancy of existing models to new samples. We evaluate MiDAS on generalization to drifted data with 9 fake news datasets.
  arXiv  Detail & Related papers  (2022-05-19T19:36:08Z)
• Vision Transformers are Robust Learners [65.91359312429147]
  We study the robustness of the Vision Transformer (ViT) against common corruptions and perturbations, distribution shifts, and natural adversarial examples. We present analyses that provide both quantitative and qualitative indications to explain why ViTs are indeed more robust learners.
  arXiv  Detail & Related papers  (2021-05-17T02:39:22Z)
• Pre-trained Summarization Distillation [121.14806854092672]
  Recent work on distilling BERT for classification and regression tasks shows strong performance using direct knowledge distillation. Alternatively, machine translation practitioners distill using pseudo-labeling, where a small model is trained on the translations of a larger model. A third, simpler approach is to'shrink and fine-tune' (SFT), which avoids any explicit distillation by copying parameters to a smaller student model and then fine-tuning.
  arXiv  Detail & Related papers  (2020-10-24T23:15:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.