Attribute-Based Diagnosis of LLM Alignment with Hate Speech Annotations

• URL: http://arxiv.org/abs/2605.27025v1
• Date: Tue, 26 May 2026 13:44:48 GMT
• Title: Attribute-Based Diagnosis of LLM Alignment with Hate Speech Annotations
• Abstract summary: Hate speech annotation is costly, subjective, and prone to annotator disagreement.<n>We analyze how well large language models (LLMs) align with human judgments.<n>We reconstruct continuous hate speech scores from the Measuring Hate Speech corpus.
• Score: 48.69228180369574
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Hate speech annotation is costly, subjective, and prone to annotator disagreement, making large-scale dataset construction challenging. We systematically analyze how well large language models (LLMs) align with human judgments across ten theoretically grounded subjective attributes, such as dehumanization, violence, and sentiment, evaluating both small and large variants of Llama 3.1 and Qwen 2.5. Our analysis reveals a consistent split across all models: behaviorally explicit dimensions (insult, humiliate, attack-defend) correlate strongly with human annotations, while evaluative dimensions (respect, sentiment, hate speech) are systematically inverted. Demographic persona conditioning reduces model confidence without improving alignment. Building on these insights, we propose combining attribute-level LLM predictions via a confidence-weighted Ridge regression to reconstruct continuous hate speech scores from the Measuring Hate Speech corpus, achieving $R^2$ of up to 0.71 and outperforming direct prompting baselines, demonstrating that structured attribute decomposition recovers a richer and more human-aligned signal than end-to-end label prediction alone.

Related papers

• Whitewashing Hate, Smearing Harmless Content: Annotator-Style Rebuttal Attacks on LLM-Based Moderation [46.54749884988069]
  Large language models (LLMs) are increasingly used for hate speech moderation, often within human-AI.<n>This study examines the susceptibility of initially correct model judgments to annotator-style rebuttals.<n>We introduce a rejudge protocol that extends direct contradiction with decision-boundary perturbations and adversarial rationales.
  arXiv  Detail & Related papers  (2026-08-23T05:59:25Z)
• The Ghost Annotator: a Framework to Explore Human Label Variation in Content Moderation through Conformal Prediction [39.10305114526259]
  We introduce a framework combining conformal prediction with Collaborative Filtering-style annotators' representation.<n>Using Non-Conformity Scores, we introduce the Ghost Prediction metric and the Ghost Annotator representation to quantify cases in which model predictions diverge from all available human annotations.<n>Our finding shows that while all models uncertainty increases with annotator disagreement, larger models tend to be more confident in the classification of texts that are not aligned with any human annotation.
  arXiv  Detail & Related papers  (2026-06-01T21:32:37Z)
• Mastering Negation: Boosting Grounding Models via Grouped Opposition-Based Learning [87.15765427638195]
  We introduce D-Negation, a new dataset that provides objects annotated with both positive and negative semantic descriptions.<n>We propose a grouped opposition-based learning framework that learns negation-aware representations from limited samples.<n>By fine-tuning fewer than 10 percent of the model parameters, our approach achieves improvements of up to 4.4 mAP and 5.7 mAP on positive and negative semantic evaluations.
  arXiv  Detail & Related papers  (2026-03-13T03:21:48Z)
• Human Supervision as an Information Bottleneck: A Unified Theory of Error Floors in Human-Guided Learning [51.56484100374058]
  We argue that limitations reflect structural properties of the supervision channel rather than model scale or optimization.<n>We develop a unified theory showing that whenever the human supervision channel is not sufficient for a latent evaluation target, it acts as an information-reducing channel.
  arXiv  Detail & Related papers  (2026-02-26T19:11:32Z)
• Understanding Degradation with Vision Language Model [56.09241449206817]
  Understanding visual degradations is a critical yet challenging problem in computer vision.<n>We introduce DU-VLM, a multimodal chain-of-thought model trained with supervised fine-tuning and reinforcement learning.<n>We also introduce textbfDU-110k, a large-scale dataset comprising 110,000 clean-degraded pairs with grounded physical annotations.
  arXiv  Detail & Related papers  (2026-02-04T13:51:15Z)
• Boosting Accuracy and Interpretability in Multilingual Hate Speech Detection Through Layer Freezing and Explainable AI [0.0]
  This study examines the performance of three transformer-based models for sentiment analysis and hate speech detection.<n>The evaluation is conducted across five languages: English, Korean, Japanese, Chinese, and French.<n>We integrate the Local Interpretable Model-agnostic Explanations (LIME) framework to highlight the contribution of individual words to the models decisions.
  arXiv  Detail & Related papers  (2026-01-06T04:07:51Z)
• Measuring the Unspoken: A Disentanglement Model and Benchmark for Psychological Analysis in the Wild [19.78493693965451]
  Multilevel Insight Network for Disentanglement(MIND) is a novel hierarchical visual encoder.<n>ConvoInsight-DB is a new large-scale dataset with expert annotations for micro-expressions and deep psychological inference.<n>On our PRISM benchmark, MIND significantly outperforms all baselines, achieving a +86.95% gain in micro-expression detection over prior SOTA.
  arXiv  Detail & Related papers  (2025-12-04T12:13:18Z)
• Can Reasoning Help Large Language Models Capture Human Annotator Disagreement? [84.32752330104775]
  Variation in human annotation (i.e., disagreements) is common in NLP.<n>We evaluate the influence of different reasoning settings on Large Language Model disagreement modeling.<n>Surprisingly, our results show that RLVR-style reasoning degrades performance in disagreement modeling.
  arXiv  Detail & Related papers  (2025-06-24T09:49:26Z)
• Human and LLM Biases in Hate Speech Annotations: A Socio-Demographic Analysis of Annotators and Targets [0.6918368994425961]
  We leverage an extensive dataset with rich socio-demographic information of both annotators and targets.<n>Our analysis surfaces the presence of widespread biases, which we quantitatively describe and characterize based on their intensity and prevalence.<n>Our work offers new and nuanced results on human biases in hate speech annotations, as well as fresh insights into the design of AI-driven hate speech detection systems.
  arXiv  Detail & Related papers  (2024-10-10T14:48:57Z)
• Constructing interval variables via faceted Rasch measurement and multitask deep learning: a hate speech application [63.10266319378212]
  We propose a method for measuring complex variables on a continuous, interval spectrum by combining supervised deep learning with the Constructing Measures approach to faceted Rasch item response theory (IRT) We demonstrate this new method on a dataset of 50,000 social media comments sourced from YouTube, Twitter, and Reddit and labeled by 11,000 U.S.-based Amazon Mechanical Turk workers.
  arXiv  Detail & Related papers  (2020-09-22T02:15:05Z)
• Characterizing Speech Adversarial Examples Using Self-Attention U-Net Enhancement [102.48582597586233]
  We present a U-Net based attention model, U-Net$_At$, to enhance adversarial speech signals. We conduct experiments on the automatic speech recognition (ASR) task with adversarial audio attacks.
  arXiv  Detail & Related papers  (2020-03-31T02:16:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.