Offensive Language Detection with BERT-based models, By Customizing Attention Probabilities

• URL: http://arxiv.org/abs/2110.05133v1
• Date: Mon, 11 Oct 2021 10:23:44 GMT
• Title: Offensive Language Detection with BERT-based models, By Customizing Attention Probabilities
• Authors: Peyman Alavi, Pouria Nikvand, Mehrnoush Shamsfard
• Abstract summary: We suggest a methodology to enhance the performance of the BERT-based models on the Offensive Language Detection' task. We customize attention probabilities by changing the Attention Mask' input to create more efficacious word embeddings. The most improvement was 2% and 10% for English and Persian languages, respectively.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper describes a novel study on using `Attention Mask' input in transformers and using this approach for detecting offensive content in both English and Persian languages. The paper's principal focus is to suggest a methodology to enhance the performance of the BERT-based models on the `Offensive Language Detection' task. Therefore, we customize attention probabilities by changing the `Attention Mask' input to create more efficacious word embeddings. To do this, we firstly tokenize the training set of the exploited datasets (by BERT tokenizer). Then, we apply Multinomial Naive Bayes to map these tokens to two probabilities. These probabilities indicate the likelihood of making a text non-offensive or offensive, provided that it contains that token. Afterwards, we use these probabilities to define a new term, namely Offensive Score. Next, we create two separate (because of the differences in the types of the employed datasets) equations based on Offensive Scores for each language to re-distribute the `Attention Mask' input for paying more attention to more offensive phrases. Eventually, we put the F1-macro score as our evaluation metric and fine-tune several combinations of BERT with ANNs, CNNs and RNNs to examine the effect of using this methodology on various combinations. The results indicate that all models will enhance with this methodology. The most improvement was 2% and 10% for English and Persian languages, respectively.

Related papers

• Beyond Image-Text Matching: Verb Understanding in Multimodal Transformers Using Guided Masking [0.4543820534430524]
  This work introduces an alternative probing strategy called guided masking. The proposed approach ablates different modalities using masking and assesses the model's ability to predict the masked word with high accuracy. We show that guided masking on ViLBERT, LXMERT, UNITER, and VisualBERT can predict the correct verb with high accuracy.
  arXiv  Detail & Related papers  (2024-01-29T21:22:23Z)
• Muted: Multilingual Targeted Offensive Speech Identification and Visualization [15.656203119337436]
  Muted is a system to identify multilingual HAP content by displaying offensive arguments and their targets using heat maps to indicate their intensity. We present the model's performance on identifying offensive spans and their targets in existing datasets and present new annotations on German text.
  arXiv  Detail & Related papers  (2023-12-18T16:50:27Z)
• Object Recognition as Next Token Prediction [99.40793702627396]
  We present an approach to pose object recognition as next token prediction. The idea is to apply a language decoder that auto-regressively predicts the text tokens from image embeddings to form labels.
  arXiv  Detail & Related papers  (2023-12-04T18:58:40Z)
• TagCLIP: Improving Discrimination Ability of Open-Vocabulary Semantic Segmentation [53.974228542090046]
  Contrastive Language-Image Pre-training (CLIP) has recently shown great promise in pixel-level zero-shot learning tasks. Existing approaches utilizing CLIP's text and patch embeddings to generate semantic masks often misidentify input pixels from unseen classes. We propose TagCLIP (Trusty-aware guided CLIP) to address this issue.
  arXiv  Detail & Related papers  (2023-04-15T12:52:23Z)
• Ensemble Transfer Learning for Multilingual Coreference Resolution [60.409789753164944]
  A problem that frequently occurs when working with a non-English language is the scarcity of annotated training data. We design a simple but effective ensemble-based framework that combines various transfer learning techniques. We also propose a low-cost TL method that bootstraps coreference resolution models by utilizing Wikipedia anchor texts.
  arXiv  Detail & Related papers  (2023-01-22T18:22:55Z)
• AWTE-BERT:Attending to Wordpiece Tokenization Explicitly on BERT for Joint Intent Classification and SlotFilling [5.684659127683238]
  BERT (Bidirectional Representations from Transformers) achieves the joint optimization of the two tasks. We propose a novel joint model based on BERT, which explicitly models the multiple sub-tokens features after wordpiece tokenization. Experimental results demonstrate that our proposed model achieves significant improvement on intent classification accuracy, slot filling F1, and sentence-level semantic frame accuracy.
  arXiv  Detail & Related papers  (2022-11-27T13:49:19Z)
• A new approach to calculating BERTScore for automatic assessment of translation quality [0.0]
  The study focuses on the applicability of the BERTScore metric to translation quality assessment at the sentence level for English -> Russian direction. Experiments were performed with a pre-trained Multilingual BERT as well as with a pair of Monolingual BERT models. It was demonstrated that such transformation helps to prevent mismatching issue and shown that this approach gives better results than using embeddings of the Multilingual model.
  arXiv  Detail & Related papers  (2022-03-10T19:25:16Z)
• Revisiting Self-Training for Few-Shot Learning of Language Model [61.173976954360334]
  Unlabeled data carry rich task-relevant information, they are proven useful for few-shot learning of language model. In this work, we revisit the self-training technique for language model fine-tuning and present a state-of-the-art prompt-based few-shot learner, SFLM.
  arXiv  Detail & Related papers  (2021-10-04T08:51:36Z)
• Using BERT Encoding to Tackle the Mad-lib Attack in SMS Spam Detection [0.0]
  We investigate whether language models sensitive to the semantics and context of words, such as Google's BERT, may be useful to overcome this adversarial attack. Using a dataset of 5572 SMS spam messages, we first established a baseline of detection performance. Then, we built a thesaurus of the vocabulary contained in these messages, and set up a Mad-lib attack experiment. We found that the classic models achieved a 94% Balanced Accuracy (BA) in the original dataset, whereas the BERT model obtained 96%.
  arXiv  Detail & Related papers  (2021-07-13T21:17:57Z)
• Leveraging Adversarial Training in Self-Learning for Cross-Lingual Text Classification [52.69730591919885]
  We present a semi-supervised adversarial training process that minimizes the maximal loss for label-preserving input perturbations. We observe significant gains in effectiveness on document and intent classification for a diverse set of languages.
  arXiv  Detail & Related papers  (2020-07-29T19:38:35Z)
• BERT-ATTACK: Adversarial Attack Against BERT Using BERT [77.82947768158132]
  Adrial attacks for discrete data (such as texts) are more challenging than continuous data (such as images) We propose textbfBERT-Attack, a high-quality and effective method to generate adversarial samples. Our method outperforms state-of-the-art attack strategies in both success rate and perturb percentage.
  arXiv  Detail & Related papers  (2020-04-21T13:30:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.