Contextualized Counterspeech: Strategies for Adaptation, Personalization, and Evaluation

• URL: http://arxiv.org/abs/2412.07338v3
• Date: Fri, 07 Feb 2025 10:30:23 GMT
• Title: Contextualized Counterspeech: Strategies for Adaptation, Personalization, and Evaluation
• Authors: Lorenzo Cima, Alessio Miaschi, Amaury Trujillo, Marco Avvenuti, Felice Dell'Orletta, Stefano Cresci,
• Abstract summary: We propose and evaluate strategies for generating tailored counterspeech that is adapted to the moderation context and personalized for the moderated user. Results show that contextualized counterspeech can significantly outperform state-of-the-art generic counterspeech in adequacy and persuasiveness. The effectiveness of contextualized AI-generated counterspeech and the divergence between human and algorithmic evaluations underscore the importance of increased human-AI collaboration in content moderation.
• Score: 2.1944577276732726
• License:
• Abstract: AI-generated counterspeech offers a promising and scalable strategy to curb online toxicity through direct replies that promote civil discourse. However, current counterspeech is one-size-fits-all, lacking adaptation to the moderation context and the users involved. We propose and evaluate multiple strategies for generating tailored counterspeech that is adapted to the moderation context and personalized for the moderated user. We instruct an LLaMA2-13B model to generate counterspeech, experimenting with various configurations based on different contextual information and fine-tuning strategies. We identify the configurations that generate persuasive counterspeech through a combination of quantitative indicators and human evaluations collected via a pre-registered mixed-design crowdsourcing experiment. Results show that contextualized counterspeech can significantly outperform state-of-the-art generic counterspeech in adequacy and persuasiveness, without compromising other characteristics. Our findings also reveal a poor correlation between quantitative indicators and human evaluations, suggesting that these methods assess different aspects and highlighting the need for nuanced evaluation methodologies. The effectiveness of contextualized AI-generated counterspeech and the divergence between human and algorithmic evaluations underscore the importance of increased human-AI collaboration in content moderation.

Related papers

• CSEval: Towards Automated, Multi-Dimensional, and Reference-Free Counterspeech Evaluation using Auto-Calibrated LLMs [18.827745815939213]
  We introduce CSEval, a novel dataset and framework for evaluating counterspeech quality across four dimensions. We propose Auto-Calibrated COT for Counterspeech Evaluation (Auto-CSEval), a prompt-based method with auto-calibrated chain-of-thoughts. Our experiments show that Auto-CSEval outperforms traditional metrics like ROUGE, METEOR, and BertScore in correlating with human judgement.
  arXiv  Detail & Related papers  (2025-01-29T11:38:29Z)
• Assessing the Human Likeness of AI-Generated Counterspeech [10.434435022492723]
  This paper investigates the human likeness of AI-generated counterspeech. We implement and evaluate several LLM-based generation strategies. We reveal differences in linguistic characteristics, politeness, and specificity.
  arXiv  Detail & Related papers  (2024-10-14T18:48:47Z)
• Predictive Speech Recognition and End-of-Utterance Detection Towards Spoken Dialog Systems [55.99999020778169]
  We study a function that can predict the forthcoming words and estimate the time remaining until the end of an utterance. We develop a cross-attention-based algorithm that incorporates both acoustic and linguistic information. Results demonstrate the proposed model's ability to predict upcoming words and estimate future EOU events up to 300ms prior to the actual EOU.
  arXiv  Detail & Related papers  (2024-09-30T06:29:58Z)
• Outcome-Constrained Large Language Models for Countering Hate Speech [10.434435022492723]
  This study aims to develop methods for generating counterspeech constrained by conversation outcomes. We experiment with large language models (LLMs) to incorporate into the text generation process two desired conversation outcomes. Evaluation results show that our methods effectively steer the generation of counterspeech toward the desired outcomes.
  arXiv  Detail & Related papers  (2024-03-25T19:44:06Z)
• A Multi-Aspect Framework for Counter Narrative Evaluation using Large Language Models [16.878541623617473]
  Counter narratives are informed responses to hate speech contexts designed to refute hateful claims and de-escalate encounters. Previous automatic metrics for counter narrative evaluation lack alignment with human judgment. We propose a novel evaluation framework prompting LLMs to provide scores and feedback for generated counter narrative candidates.
  arXiv  Detail & Related papers  (2024-02-18T18:56:07Z)
• Automatically measuring speech fluency in people with aphasia: first achievements using read-speech data [55.84746218227712]
  This study aims at assessing the relevance of a signalprocessingalgorithm, initially developed in the field of language acquisition, for the automatic measurement of speech fluency.
  arXiv  Detail & Related papers  (2023-08-09T07:51:40Z)
• On the Behavior of Intrusive and Non-intrusive Speech Enhancement Metrics in Predictive and Generative Settings [14.734454356396157]
  We evaluate the performance of the same speech enhancement backbone trained under predictive and generative paradigms. We show that intrusive and non-intrusive measures correlate differently for each paradigm.
  arXiv  Detail & Related papers  (2023-06-05T16:30:17Z)
• Towards Quantifiable Dialogue Coherence Evaluation [126.55560816209756]
  Quantifiable Dialogue Coherence Evaluation (QuantiDCE) is a novel framework aiming to train a quantifiable dialogue coherence metric. QuantiDCE includes two training stages, Multi-Level Ranking (MLR) pre-training and Knowledge Distillation (KD) fine-tuning. Experimental results show that the model trained by QuantiDCE presents stronger correlations with human judgements than the other state-of-the-art metrics.
  arXiv  Detail & Related papers  (2021-06-01T14:11:17Z)
• Towards Robust Speech-to-Text Adversarial Attack [78.5097679815944]
  This paper introduces a novel adversarial algorithm for attacking the state-of-the-art speech-to-text systems, namely DeepSpeech, Kaldi, and Lingvo. Our approach is based on developing an extension for the conventional distortion condition of the adversarial optimization formulation. Minimizing over this metric, which measures the discrepancies between original and adversarial samples' distributions, contributes to crafting signals very close to the subspace of legitimate speech recordings.
  arXiv  Detail & Related papers  (2021-03-15T01:51:41Z)
• Is this Dialogue Coherent? Learning from Dialogue Acts and Entities [82.44143808977209]
  We create the Switchboard Coherence (SWBD-Coh) corpus, a dataset of human-human spoken dialogues annotated with turn coherence ratings. Our statistical analysis of the corpus indicates how turn coherence perception is affected by patterns of distribution of entities. We find that models combining both DA and entity information yield the best performances both for response selection and turn coherence rating.
  arXiv  Detail & Related papers  (2020-06-17T21:02:40Z)
• You Impress Me: Dialogue Generation via Mutual Persona Perception [62.89449096369027]
  The research in cognitive science suggests that understanding is an essential signal for a high-quality chit-chat conversation. Motivated by this, we propose P2 Bot, a transmitter-receiver based framework with the aim of explicitly modeling understanding.
  arXiv  Detail & Related papers  (2020-04-11T12:51:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.