Ruddit: Norms of Offensiveness for English Reddit Comments

• URL: http://arxiv.org/abs/2106.05664v2
• Date: Fri, 11 Jun 2021 07:41:58 GMT
• Title: Ruddit: Norms of Offensiveness for English Reddit Comments
• Authors: Rishav Hada, Sohi Sudhir, Pushkar Mishra, Helen Yannakoudakis, Saif M. Mohammad, Ekaterina Shutova
• Abstract summary: We create the first dataset of English language Reddit comments that has fine-grained, real-valued scores between -1 and 1. We show that the method produces highly reliable offensiveness scores. We evaluate the ability of widely-used neural models to predict offensiveness scores on this new dataset.
• Score: 35.83156813452207
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On social media platforms, hateful and offensive language negatively impact the mental well-being of users and the participation of people from diverse backgrounds. Automatic methods to detect offensive language have largely relied on datasets with categorical labels. However, comments can vary in their degree of offensiveness. We create the first dataset of English language Reddit comments that has fine-grained, real-valued scores between -1 (maximally supportive) and 1 (maximally offensive). The dataset was annotated using Best--Worst Scaling, a form of comparative annotation that has been shown to alleviate known biases of using rating scales. We show that the method produces highly reliable offensiveness scores. Finally, we evaluate the ability of widely-used neural models to predict offensiveness scores on this new dataset.

Related papers

• OffensiveLang: A Community Based Implicit Offensive Language Dataset [5.813922783967869]
  Hate speech or offensive languages exist in both explicit and implicit forms. OffensiveLang is a community based implicit offensive language dataset. We present a prompt-based approach that effectively generates implicit offensive languages.
  arXiv  Detail & Related papers  (2024-03-04T20:34:58Z)
• The Impact of Debiasing on the Performance of Language Models in Downstream Tasks is Underestimated [70.23064111640132]
  We compare the impact of debiasing on performance across multiple downstream tasks using a wide-range of benchmark datasets. Experiments show that the effects of debiasing are consistently emphunderestimated across all tasks.
  arXiv  Detail & Related papers  (2023-09-16T20:25:34Z)
• Just Rank: Rethinking Evaluation with Word and Sentence Similarities [105.5541653811528]
  intrinsic evaluation for embeddings lags far behind, and there has been no significant update since the past decade. This paper first points out the problems using semantic similarity as the gold standard for word and sentence embedding evaluations. We propose a new intrinsic evaluation method called EvalRank, which shows a much stronger correlation with downstream tasks.
  arXiv  Detail & Related papers  (2022-03-05T08:40:05Z)
• Detecting Offensive Language on Social Networks: An End-to-end Detection Method based on Graph Attention Networks [7.723697303436006]
  We propose an end-to-end method based on community structure and text features for offensive language detection (CT-OLD) We add user opinion to the community structure for representing user features. The user opinion is represented by user historical behavior information, which outperforms that represented by text information.
  arXiv  Detail & Related papers  (2022-03-04T03:57:18Z)
• Detecting Abusive Albanian [5.092028049119383]
  scShaj is an annotated dataset for hate speech and offensive speech constructed from user-text content on various social media platforms. The dataset is tested using three different classification models, the best of which achieves an F1 score of 0.77 for the identification of offensive language.
  arXiv  Detail & Related papers  (2021-07-28T18:47:32Z)
• Double Perturbation: On the Robustness of Robustness and Counterfactual Bias Evaluation [109.06060143938052]
  We propose a "double perturbation" framework to uncover model weaknesses beyond the test dataset. We apply this framework to study two perturbation-based approaches that are used to analyze models' robustness and counterfactual bias in English.
  arXiv  Detail & Related papers  (2021-04-12T06:57:36Z)
• Weakly-Supervised Aspect-Based Sentiment Analysis via Joint Aspect-Sentiment Topic Embedding [71.2260967797055]
  We propose a weakly-supervised approach for aspect-based sentiment analysis. We learn sentiment, aspect> joint topic embeddings in the word embedding space. We then use neural models to generalize the word-level discriminative information.
  arXiv  Detail & Related papers  (2020-10-13T21:33:24Z)
• Trawling for Trolling: A Dataset [56.1778095945542]
  We present a dataset that models trolling as a subcategory of offensive content. The dataset has 12,490 samples, split across 5 classes; Normal, Profanity, Trolling, Derogatory and Hate Speech.
  arXiv  Detail & Related papers  (2020-08-02T17:23:55Z)
• Arabic Offensive Language on Twitter: Analysis and Experiments [9.879488163141813]
  We introduce a method for building a dataset that is not biased by topic, dialect, or target. We produce the largest Arabic dataset to date with special tags for vulgarity and hate speech.
  arXiv  Detail & Related papers  (2020-04-05T13:05:11Z)
• Offensive Language Detection: A Comparative Analysis [2.5739449801033842]
  We explore the effectiveness of Google sentence encoder, Fasttext, Dynamic mode decomposition (DMD) based features and Random kitchen sink (RKS) method for offensive language detection. From the experiments and evaluation we observed that RKS with fastetxt achieved competing results.
  arXiv  Detail & Related papers  (2020-01-09T17:48:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.