Evaluating Large Language Models for Antisemitic Incident Classification

• URL: http://arxiv.org/abs/2607.04890v1
• Date: Mon, 06 Jul 2026 10:14:53 GMT
• Title: Evaluating Large Language Models for Antisemitic Incident Classification
• Abstract summary: We introduce the task of hateful event detection and investigate the ability of AI systems to discover and classify reports of antisemitic events with fine-grained labels.<n>We evaluate OpenAI's GPT-4o and Meta's Llama-3.2-3B-Instruct on datasets containing antisemitic event descriptions from news articles, civil society reports, and official records.
• Score: 64.21103964951531
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Addressing hate and violence in society requires timely detection of hateful events from public reporting, but automated identification of hateful events remains underexplored. We introduce the task of hateful event detection and investigate the ability of AI systems, specifically large language models (LLMs), to discover and classify reports of antisemitic events with fine-grained labels. We evaluate OpenAI's GPT-4o and Meta's Llama-3.2-3B-Instruct on multiple expert-annotated datasets containing antisemitic event descriptions from news articles, civil society reports, and official records. We show that LLMs, particularly GPT-4o, have potential for this task, but substantial improvement is needed. Providing clear term definitions and in-context examples in prompts can improve performance: definitions are most helpful for rhetoric-oriented events (e.g. classical antisemitic tropes), while examples help label action-oriented events (e.g. physical assault). A case study of college newspapers demonstrates that LLMs can help surface relevant real-world events, supporting early monitoring and intervention. Overall, our findings highlight both opportunities and critical gaps in AI's ability to recognize complex harms and underscore the need for collaborative efforts among AI developers, policymakers, and civil society to design models, implement robust evaluation, and develop policy frameworks for defining and combating hate efficiently and effectively.

Related papers

• Impacts of Racial Bias in Historical Training Data for News AI [0.0]
  This paper investigates one such example trained on the broadly-used New York Times Annotated Corpus to create a multi-label classifier.<n>We find that the "blacks" label operates partially as a general "racism detector" across some minoritized groups.<n>However, it performs poorly against expectations on modern examples such as COVID-19 era anti-Asian hate stories, and reporting on the Black Lives Matter movement.
  arXiv  Detail & Related papers  (2025-12-18T18:56:11Z)
• Self-evolving expertise in complex non-verifiable subject domains: dialogue as implicit meta-RL [0.0]
  So-called wicked problems', those involving complex multi-dimensional settings, non-verifiable outcomes, heterogeneous impacts and a lack of single objectively correct answers, have plagued humans throughout history.<n>The use of state-of-the-art artificial intelligence systems (notably Large Language Model-based agents) collaborating with humans on solving such problems is being actively explored.<n>This work address this gap with Dialectica, a framework where agents engage in structured dialogue on defined topics, augmented by memory, self-reflection, and policy-constrained context editing.
  arXiv  Detail & Related papers  (2025-10-17T15:59:44Z)
• Teaching Language Models To Gather Information Proactively [53.85419549904644]
  Large language models (LLMs) are increasingly expected to function as collaborative partners.<n>In this work, we introduce a new task paradigm: proactive information gathering.<n>We design a scalable framework that generates partially specified, real-world tasks, masking key information.<n>Within this setup, our core innovation is a reinforcement finetuning strategy that rewards questions that elicit genuinely new, implicit user information.
  arXiv  Detail & Related papers  (2025-07-28T23:50:09Z)
• From Incidents to Insights: Patterns of Responsibility following AI Harms [1.9389881806157316]
  The AI Incident Database was inspired by aviation safety databases, which enable collective learning from failures to prevent future incidents.<n>The database documents hundreds of AI failures, collected from the news and media.<n>We argue that by looking beyond technically-focused learning, the dataset can provide new, highly valuable insights.
  arXiv  Detail & Related papers  (2025-05-07T09:59:36Z)
• Contextualize-then-Aggregate: Circuits for In-Context Learning in Gemma-2 2B [51.74607395697567]
  In-Context Learning (ICL) is an intriguing ability of large language models (LLMs)<n>We use causal interventions to identify information flow in Gemma-2 2B for five naturalistic ICL tasks.<n>We find that the model infers task information using a two-step strategy we call contextualize-then-aggregate.
  arXiv  Detail & Related papers  (2025-03-31T18:33:55Z)
• From Objects to Events: Unlocking Complex Visual Understanding in Object Detectors via LLM-guided Symbolic Reasoning [71.41062111470414]
  Current object detectors excel at entity localization and classification, yet exhibit inherent limitations in event recognition capabilities.<n>We present a novel framework that expands the capability of standard object detectors beyond mere object recognition to complex event understanding.<n>Our key innovation lies in bridging the semantic gap between object detection and event understanding without requiring expensive task-specific training.
  arXiv  Detail & Related papers  (2025-02-09T10:30:54Z)
• Investigating Annotator Bias in Large Language Models for Hate Speech Detection [5.589665886212444]
  This paper delves into the biases present in Large Language Models (LLMs) when annotating hate speech data. Specifically targeting highly vulnerable groups within these categories, we analyze annotator biases. We introduce our custom hate speech detection dataset, HateBiasNet, to conduct this research.
  arXiv  Detail & Related papers  (2024-06-17T00:18:31Z)
• Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks [5.860289498416911]
  Large Language Models (LLMs) are swiftly advancing in architecture and capability. As they integrate more deeply into complex systems, the urgency to scrutinize their security properties grows. This paper surveys research in the emerging interdisciplinary field of adversarial attacks on LLMs.
  arXiv  Detail & Related papers  (2023-10-16T21:37:24Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)
• Predicting Themes within Complex Unstructured Texts: A Case Study on Safeguarding Reports [66.39150945184683]
  We focus on the problem of automatically identifying the main themes in a safeguarding report using supervised classification approaches. Our results show the potential of deep learning models to simulate subject-expert behaviour even for complex tasks with limited labelled data.
  arXiv  Detail & Related papers  (2020-10-27T19:48:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.