Evaluating ChatGPT's Performance for Multilingual and Emoji-based Hate Speech Detection

• URL: http://arxiv.org/abs/2305.13276v2
• Date: Tue, 23 May 2023 03:39:44 GMT
• Title: Evaluating ChatGPT's Performance for Multilingual and Emoji-based Hate Speech Detection
• Authors: Mithun Das, Saurabh Kumar Pandey, Animesh Mukherjee
• Abstract summary: Large language models like ChatGPT have recently shown a great promise in performing several tasks, including hate speech detection. This study aims to evaluate the strengths and weaknesses of the ChatGPT model in detecting hate speech at a granular level across 11 languages.
• Score: 4.809236881780707
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hate speech is a severe issue that affects many online platforms. So far, several studies have been performed to develop robust hate speech detection systems. Large language models like ChatGPT have recently shown a great promise in performing several tasks, including hate speech detection. However, it is crucial to comprehend the limitations of these models to build robust hate speech detection systems. To bridge this gap, our study aims to evaluate the strengths and weaknesses of the ChatGPT model in detecting hate speech at a granular level across 11 languages. Our evaluation employs a series of functionality tests that reveals various intricate failures of the model which the aggregate metrics like macro F1 or accuracy are not able to unfold. In addition, we investigate the influence of complex emotions, such as the use of emojis in hate speech, on the performance of the ChatGPT model. Our analysis highlights the shortcomings of the generative models in detecting certain types of hate speech and highlighting the need for further research and improvements in the workings of these models.

Related papers

• Exploring Large Language Models for Hate Speech Detection in Rioplatense Spanish [0.08192907805418582]
  Hate speech detection deals with many language variants, slang, slurs, expression modalities, and cultural nuances. This work presents a brief analysis of the performance of large language models in the detection of Hate Speech for Rioplatense Spanish.
  arXiv  Detail & Related papers  (2024-10-16T02:32:12Z)
• Bridging Modalities: Enhancing Cross-Modality Hate Speech Detection with Few-Shot In-Context Learning [4.136573141724715]
  Hate speech on the internet poses a significant challenge to digital platform safety. Recent research has developed detection models tailored to specific modalities. This study conducts extensive experiments using few-shot in-context learning with large language models.
  arXiv  Detail & Related papers  (2024-10-08T01:27:12Z)
• An Investigation of Large Language Models for Real-World Hate Speech Detection [46.15140831710683]
  A major limitation of existing methods is that hate speech detection is a highly contextual problem. Recently, large language models (LLMs) have demonstrated state-of-the-art performance in several natural language tasks. Our study reveals that a meticulously crafted reasoning prompt can effectively capture the context of hate speech.
  arXiv  Detail & Related papers  (2024-01-07T00:39:33Z)
• SpeechX: Neural Codec Language Model as a Versatile Speech Transformer [57.82364057872905]
  SpeechX is a versatile speech generation model capable of zero-shot TTS and various speech transformation tasks. Experimental results show SpeechX's efficacy in various tasks, including zero-shot TTS, noise suppression, target speaker extraction, speech removal, and speech editing with or without background noise.
  arXiv  Detail & Related papers  (2023-08-14T01:01:19Z)
• To ChatGPT, or not to ChatGPT: That is the question! [78.407861566006]
  This study provides a comprehensive and contemporary assessment of the most recent techniques in ChatGPT detection. We have curated a benchmark dataset consisting of prompts from ChatGPT and humans, including diverse questions from medical, open Q&A, and finance domains. Our evaluation results demonstrate that none of the existing methods can effectively detect ChatGPT-generated content.
  arXiv  Detail & Related papers  (2023-04-04T03:04:28Z)
• Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Models [14.128029444990895]
  We introduce HateCheck (MHC), a suite of functional tests for multilingual hate speech detection models. MHC covers 34 functionalities across ten languages, which is more languages than any other hate speech dataset. We train and test a high-performing multilingual hate speech detection model, and reveal critical model weaknesses for monolingual and cross-lingual applications.
  arXiv  Detail & Related papers  (2022-06-20T17:54:39Z)
• HateCheckHIn: Evaluating Hindi Hate Speech Detection Models [6.52974752091861]
  multilingual hate is a major emerging challenge for automated detection. We introduce a set of functionalities for the purpose of evaluation. Considering Hindi as a base language, we craft test cases for each functionality.
  arXiv  Detail & Related papers  (2022-04-30T19:09:09Z)
• Deep Learning for Hate Speech Detection: A Comparative Study [54.42226495344908]
  We present here a large-scale empirical comparison of deep and shallow hate-speech detection methods. Our goal is to illuminate progress in the area, and identify strengths and weaknesses in the current state-of-the-art. In doing so we aim to provide guidance as to the use of hate-speech detection in practice, quantify the state-of-the-art, and identify future research directions.
  arXiv  Detail & Related papers  (2022-02-19T03:48:20Z)
• Addressing the Challenges of Cross-Lingual Hate Speech Detection [115.1352779982269]
  In this paper we focus on cross-lingual transfer learning to support hate speech detection in low-resource languages. We leverage cross-lingual word embeddings to train our neural network systems on the source language and apply it to the target language. We investigate the issue of label imbalance of hate speech datasets, since the high ratio of non-hate examples compared to hate examples often leads to low model performance.
  arXiv  Detail & Related papers  (2022-01-15T20:48:14Z)
• Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units [56.52704348773307]
  We propose a novel LSTM-based generative speech LM based on linguistic units including syllables and phonemes. With a limited dataset, orders of magnitude smaller than that required by contemporary generative models, our model closely approximates babbling speech. We show the effect of training with auxiliary text LMs, multitask learning objectives, and auxiliary articulatory features.
  arXiv  Detail & Related papers  (2021-10-31T22:48:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.