Detect Hate Speech in Unseen Domains using Multi-Task Learning: A Case Study of Political Public Figures

• URL: http://arxiv.org/abs/2208.10598v1
• Date: Mon, 22 Aug 2022 21:13:38 GMT
• Title: Detect Hate Speech in Unseen Domains using Multi-Task Learning: A Case Study of Political Public Figures
• Authors: Lanqin Yuan and Marian-Andrei Rizoiu
• Abstract summary: We propose a new Multi-task Learning pipeline that utilizes MTL to train simultaneously across multiple hate speech datasets. We show strong results when examining generalization error in train-test splits and substantial improvements when predicting on previously unseen datasets. We also assemble a novel dataset, dubbed PubFigs, focusing on the problematic speech of American Public Political Figures.
• Score: 7.52579126252489
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automatic identification of hateful and abusive content is vital in combating the spread of harmful online content and its damaging effects. Most existing works evaluate models by examining the generalization error on train-test splits on hate speech datasets. These datasets often differ in their definitions and labeling criteria, leading to poor model performance when predicting across new domains and datasets. In this work, we propose a new Multi-task Learning (MTL) pipeline that utilizes MTL to train simultaneously across multiple hate speech datasets to construct a more encompassing classification model. We simulate evaluation on new previously unseen datasets by adopting a leave-one-out scheme in which we omit a target dataset from training and jointly train on the other datasets. Our results consistently outperform a large sample of existing work. We show strong results when examining generalization error in train-test splits and substantial improvements when predicting on previously unseen datasets. Furthermore, we assemble a novel dataset, dubbed PubFigs, focusing on the problematic speech of American Public Political Figures. We automatically detect problematic speech in the $305,235$ tweets in PubFigs, and we uncover insights into the posting behaviors of public figures.

Related papers

• A Target-Aware Analysis of Data Augmentation for Hate Speech Detection [3.858155067958448]
  Hate speech is one of the main threats posed by the widespread use of social networks. We investigate the possibility of augmenting existing data with generative language models, reducing target imbalance. For some hate categories such as origin, religion, and disability, hate speech classification using augmented data for training improves by more than 10% F1 over the no augmentation baseline.
  arXiv  Detail & Related papers  (2024-10-10T15:46:27Z)
• Latent Feature-based Data Splits to Improve Generalisation Evaluation: A Hate Speech Detection Case Study [33.1099258648462]
  We present two split variants that reveal how models catastrophically fail on blind spots in the latent space. Our analysis suggests that there is no clear surface-level property of the data split that correlates with the decreased performance.
  arXiv  Detail & Related papers  (2023-11-16T23:49:55Z)
• Into the LAIONs Den: Investigating Hate in Multimodal Datasets [67.21783778038645]
  This paper investigates the effect of scaling datasets on hateful content through a comparative audit of two datasets: LAION-400M and LAION-2B. We found that hate content increased by nearly 12% with dataset scale, measured both qualitatively and quantitatively. We also found that filtering dataset contents based on Not Safe For Work (NSFW) values calculated based on images alone does not exclude all the harmful content in alt-text.
  arXiv  Detail & Related papers  (2023-11-06T19:00:05Z)
• LLMs and Finetuning: Benchmarking cross-domain performance for hate speech detection [10.014248704653]
  This study investigates the effectiveness and adaptability of pre-trained and fine-tuned Large Language Models (LLMs) in identifying hate speech. LLMs offer a huge advantage over the state-of-the-art even without pretraining. We conclude with a vision for the future of hate speech detection, emphasizing cross-domain generalizability.
  arXiv  Detail & Related papers  (2023-10-29T10:07:32Z)
• Unsupervised Sentiment Analysis of Plastic Surgery Social Media Posts [91.3755431537592]
  The massive collection of user posts across social media platforms is primarily untapped for artificial intelligence (AI) use cases. Natural language processing (NLP) is a subfield of AI that leverages bodies of documents, known as corpora, to train computers in human-like language understanding. This study demonstrates that the applied results of unsupervised analysis allow a computer to predict either negative, positive, or neutral user sentiment towards plastic surgery.
  arXiv  Detail & Related papers  (2023-07-05T20:16:20Z)
• Robust Hate Speech Detection in Social Media: A Cross-Dataset Empirical Evaluation [5.16706940452805]
  We perform a large-scale cross-dataset comparison where we fine-tune language models on different hate speech detection datasets. This analysis shows how some datasets are more generalisable than others when used as training data. Experiments show how combining hate speech detection datasets can contribute to the development of robust hate speech detection models.
  arXiv  Detail & Related papers  (2023-07-04T12:22:40Z)
• How to Solve Few-Shot Abusive Content Detection Using the Data We Actually Have [58.23138483086277]
  In this work we leverage datasets we already have, covering a wide range of tasks related to abusive language detection. Our goal is to build models cheaply for a new target label set and/or language, using only a few training examples of the target domain. Our experiments show that using already existing datasets and only a few-shots of the target task the performance of models improve both monolingually and across languages.
  arXiv  Detail & Related papers  (2023-05-23T14:04:12Z)
• BERT-based Ensemble Approaches for Hate Speech Detection [1.8734449181723825]
  This paper focuses on classifying hate speech in social media using multiple deep models. We evaluated with several ensemble techniques, including soft voting, maximum value, hard voting and stacking. Experiments have shown good results especially the ensemble models, where stacking gave F1 score of 97% on Davidson dataset and aggregating ensembles 77% on the DHO dataset.
  arXiv  Detail & Related papers  (2022-09-14T09:08:24Z)
• CHALLENGER: Training with Attribution Maps [63.736435657236505]
  We show that utilizing attribution maps for training neural networks can improve regularization of models and thus increase performance. In particular, we show that our generic domain-independent approach yields state-of-the-art results in vision, natural language processing and on time series tasks.
  arXiv  Detail & Related papers  (2022-05-30T13:34:46Z)
• The Change that Matters in Discourse Parsing: Estimating the Impact of Domain Shift on Parser Error [14.566990078034241]
  We use a statistic from the theoretical domain adaptation literature which can be directly tied to error-gap. We study the bias of this statistic as an estimator of error-gap both theoretically and through a large-scale empirical study of over 2400 experiments on 6 discourse datasets.
  arXiv  Detail & Related papers  (2022-03-21T20:04:23Z)
• A Closer Look at Debiased Temporal Sentence Grounding in Videos: Dataset, Metric, and Approach [53.727460222955266]
  Temporal Sentence Grounding in Videos (TSGV) aims to ground a natural language sentence in an untrimmed video. Recent studies have found that current benchmark datasets may have obvious moment annotation biases. We introduce a new evaluation metric "dR@n,IoU@m" that discounts the basic recall scores to alleviate the inflating evaluation caused by biased datasets.
  arXiv  Detail & Related papers  (2022-03-10T08:58:18Z)
• Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching [53.27673119360868]
  Referring expression grounding is an important and challenging task in computer vision. We propose a novel bidirectional cross-modal matching (BiCM) framework to address these challenges. Our framework outperforms previous works by 6.55% and 9.94% on two popular grounding datasets.
  arXiv  Detail & Related papers  (2022-01-18T01:13:19Z)
• Addressing the Challenges of Cross-Lingual Hate Speech Detection [115.1352779982269]
  In this paper we focus on cross-lingual transfer learning to support hate speech detection in low-resource languages. We leverage cross-lingual word embeddings to train our neural network systems on the source language and apply it to the target language. We investigate the issue of label imbalance of hate speech datasets, since the high ratio of non-hate examples compared to hate examples often leads to low model performance.
  arXiv  Detail & Related papers  (2022-01-15T20:48:14Z)
• Improving Classifier Training Efficiency for Automatic Cyberbullying Detection with Feature Density [58.64907136562178]
  We study the effectiveness of Feature Density (FD) using different linguistically-backed feature preprocessing methods. We hypothesise that estimating dataset complexity allows for the reduction of the number of required experiments. The difference in linguistic complexity of datasets allows us to additionally discuss the efficacy of linguistically-backed word preprocessing.
  arXiv  Detail & Related papers  (2021-11-02T15:48:28Z)
• Statistical Analysis of Perspective Scores on Hate Speech Detection [7.447951461558536]
  State-of-the-art hate speech classifiers are efficient only when tested on the data with the same feature distribution as training data. In such a diverse data distribution relying on low level features is the main cause of deficiency due to natural bias in data. We show that, different hate speech datasets are very similar when it comes to extract their Perspective Scores.
  arXiv  Detail & Related papers  (2021-06-22T17:17:35Z)
• On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study [65.17429512679695]
  In adversarial data collection (ADC), a human workforce interacts with a model in real time, attempting to produce examples that elicit incorrect predictions. Despite ADC's intuitive appeal, it remains unclear when training on adversarial datasets produces more robust models.
  arXiv  Detail & Related papers  (2021-06-02T00:48:33Z)
• Sentiment analysis in tweets: an assessment study from classical to modern text representation models [59.107260266206445]
  Short texts published on Twitter have earned significant attention as a rich source of information. Their inherent characteristics, such as the informal, and noisy linguistic style, remain challenging to many natural language processing (NLP) tasks. This study fulfils an assessment of existing language models in distinguishing the sentiment expressed in tweets by using a rich collection of 22 datasets.
  arXiv  Detail & Related papers  (2021-05-29T21:05:28Z)
• Competency Problems: On Finding and Removing Artifacts in Language Data [50.09608320112584]
  We argue that for complex language understanding tasks, all simple feature correlations are spurious. We theoretically analyze the difficulty of creating data for competency problems when human bias is taken into account.
  arXiv  Detail & Related papers  (2021-04-17T21:34:10Z)
• Leveraging Multi-domain, Heterogeneous Data using Deep Multitask Learning for Hate Speech Detection [21.410160004193916]
  We propose a Convolution Neural Network based multi-task learning models (MTLs)footnotecode to leverage information from multiple sources. Empirical analysis performed on three benchmark datasets shows the efficacy of the proposed approach.
  arXiv  Detail & Related papers  (2021-03-23T09:31:01Z)
• Hate Speech Detection and Racial Bias Mitigation in Social Media based on BERT model [1.9336815376402716]
  We introduce a transfer learning approach for hate speech detection based on an existing pre-trained language model called BERT. We evaluate the proposed model on two publicly available datasets annotated for racism, sexism, hate or offensive content on Twitter.
  arXiv  Detail & Related papers  (2020-08-14T16:47:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.