Wisdom of the LLM Crowd: A Large Scale Benchmark of Multi-Label U.S. Election-Related Harmful Social Media Content

• URL: http://arxiv.org/abs/2602.11962v1
• Date: Thu, 12 Feb 2026 13:57:12 GMT
• Title: Wisdom of the LLM Crowd: A Large Scale Benchmark of Multi-Label U.S. Election-Related Harmful Social Media Content
• Abstract summary: USE24-XD is a large-scale dataset of nearly 100k posts collected from X (formerly Twitter) during the 2024 U.S presidential election cycle.<n>We employ six large language models (LLMs) to systematically annotate posts across five categories: Conspiracy, Sensationalism, Hate Speech, Speculation, and Satire.
• Score: 1.7020765368200692
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The spread of election misinformation and harmful political content conveys misleading narratives and poses a serious threat to democratic integrity. Detecting harmful content at early stages is essential for understanding and potentially mitigating its downstream spread. In this study, we introduce USE24-XD, a large-scale dataset of nearly 100k posts collected from X (formerly Twitter) during the 2024 U.S. presidential election cycle, enriched with spatio-temporal metadata. To substantially reduce the cost of manual annotation while enabling scalable categorization, we employ six large language models (LLMs) to systematically annotate posts across five nuanced categories: Conspiracy, Sensationalism, Hate Speech, Speculation, and Satire. We validate LLM annotations with crowdsourcing (n = 34) and benchmark them against human annotators. Inter-rater reliability analyses show comparable agreement patterns between LLMs and humans, with LLMs exhibiting higher internal consistency and achieving up to 0.90 recall on Speculation. We apply a wisdom-of-the-crowd approach across LLMs to aggregate annotations and curate a robust multi-label dataset. 60% of posts receive at least one label. We further analyze how human annotator demographics, including political ideology and affiliation, shape labeling behavior, highlighting systematic sources of subjectivity in judgments of harmful content. The USE24-XD dataset is publicly released to support future research.

Related papers

• Contextualized Prompting For Stance Detection On Social Media [41.666445939998454]
  Stance detection on social media is challenging due to short, noisy, and context-dependent language.<n>We investigate the impact of incorporating contextual information into large language models.<n>We find that integrating contextual information improves performance, but only under specific conditions.
  arXiv  Detail & Related papers  (2026-06-04T11:14:44Z)
• StereoTales: A Multilingual Framework for Open-Ended Stereotype Discovery in LLMs [2.5649028885424374]
  StereoTales is a multilingual dataset and evaluation pipeline for studying the emergence of social bias in open-ended LLM generation.<n>The dataset covers 10 languages and 79 socio-demographic attributes, and comprises over 650k stories generated by 23 recent LLMs.<n>From these, we apply statistical tests to identify more than 1,500 over-represented associations, which we then rate for harmfulness through both a panel of humans and the same LLMs.
  arXiv  Detail & Related papers  (2026-05-11T12:12:28Z)
• MultiSoc-4D: A Benchmark for Diagnosing Instruction-Induced Label Collapse in Closed-Set LLM Annotation of Bengali Social Media [0.7437884690206271]
  We present MultiSoc-4D, a Bengali social media dataset benchmark, which contains 58K+ social media comments from six sources annotated along four dimensions: category, sentiment, hate speech, and sarcasm.<n>We discover a prevalent phenomenon called "instruction-induced label collapse", wherein LLMs show a systematic preference towards fallback labels.<n>We prove that it represents a "label illusion", statistically validated via almost null Fleiss' Kappa on sarcasm detection.
  arXiv  Detail & Related papers  (2026-05-07T20:55:28Z)
• When Hate Meets Facts: LLMs-in-the-Loop for Check-worthiness Detection in Hate Speech [4.736364625993341]
  We release WSF-ARG+, the first dataset which combines hate speech with check-worthiness information.<n>We introduce a novel LLM-in-the-loop framework to facilitate the annotation of check-worthy claims.<n>We show that HS messages with check-worthy claims show significantly higher harassment and hate.
  arXiv  Detail & Related papers  (2026-03-26T10:07:00Z)
• Latent Topic Synthesis: Leveraging LLMs for Electoral Ad Analysis [51.95395936342771]
  We introduce an end-to-end framework for automatically generating an interpretable topic taxonomy from an unlabeled corpus.<n>We apply this framework to a large corpus of Meta political ads from the month ahead of the 2024 U.S. Presidential election.<n>Our approach uncovers latent discourse structures, synthesizes semantically rich topic labels, and annotates topics with moral framing dimensions.
  arXiv  Detail & Related papers  (2025-10-16T20:30:20Z)
• Profiling News Media for Factuality and Bias Using LLMs and the Fact-Checking Methodology of Human Experts [29.95198868148809]
  We propose a novel methodology that emulates the criteria that professional fact-checkers use to assess the factuality and political bias of an entire outlet.<n>We provide an in-depth error analysis of the effect of media popularity and region on model performance.
  arXiv  Detail & Related papers  (2025-06-14T15:49:20Z)
• Search Arena: Analyzing Search-Augmented LLMs [61.28673331156436]
  We introduce Search Arena, a crowd-sourced, large-scale, human-preference dataset of over 24,000 paired multi-turn user interactions.<n>The dataset spans diverse intents and languages, and contains full system traces with around 12,000 human preference votes.<n>Our analysis reveals that user preferences are influenced by the number of citations, even when the cited content does not directly support the attributed claims.
  arXiv  Detail & Related papers  (2025-06-05T17:59:26Z)
• Analyzing Political Bias in LLMs via Target-Oriented Sentiment Classification [4.352835414206441]
  Political biases encoded by LLMs might have detrimental effects on downstream applications.<n>We propose a new approach leveraging the observation that LLM sentiment predictions vary with the target entity in the same sentence.<n>We insert 1319 demographically and politically diverse politician names in 450 political sentences and predict target-oriented sentiment using seven models in six widely spoken languages.
  arXiv  Detail & Related papers  (2025-05-26T10:01:24Z)
• Fact or Fiction? Can LLMs be Reliable Annotators for Political Truths? [2.321323878201932]
  Political misinformation poses challenges to democratic processes, shaping public opinion and trust in media. This study investigates the use of state-of-the-art large language models (LLMs) as reliable annotators for detecting political factuality in news articles.
  arXiv  Detail & Related papers  (2024-11-08T18:36:33Z)
• Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective [66.34066553400108]
  We conduct a rigorous evaluation of large language models' implicit bias towards certain demographics.<n>Inspired by psychometric principles, we propose three attack approaches, i.e., Disguise, Deception, and Teaching.<n>Our methods can elicit LLMs' inner bias more effectively than competitive baselines.
  arXiv  Detail & Related papers  (2024-06-20T06:42:08Z)
• FABLES: Evaluating faithfulness and content selection in book-length summarization [55.50680057160788]
  In this paper, we conduct the first large-scale human evaluation of faithfulness and content selection on book-length documents. We collect FABLES, a dataset of annotations on 3,158 claims made in LLM-generated summaries of 26 books, at a cost of $5.2K USD. An analysis of the annotations reveals that most unfaithful claims relate to events and character states, and they generally require indirect reasoning over the narrative to invalidate.
  arXiv  Detail & Related papers  (2024-04-01T17:33:38Z)
• Whose Side Are You On? Investigating the Political Stance of Large Language Models [56.883423489203786]
  We investigate the political orientation of Large Language Models (LLMs) across a spectrum of eight polarizing topics. Our investigation delves into the political alignment of LLMs across a spectrum of eight polarizing topics, spanning from abortion to LGBTQ issues. The findings suggest that users should be mindful when crafting queries, and exercise caution in selecting neutral prompt language.
  arXiv  Detail & Related papers  (2024-03-15T04:02:24Z)
• A Survey on Detection of LLMs-Generated Content [97.87912800179531]
  The ability to detect LLMs-generated content has become of paramount importance. We aim to provide a detailed overview of existing detection strategies and benchmarks. We also posit the necessity for a multi-faceted approach to defend against various attacks.
  arXiv  Detail & Related papers  (2023-10-24T09:10:26Z)
• Generalizing Hate Speech Detection Using Multi-Task Learning: A Case Study of Political Public Figures [3.825159708387601]
  This work proposes a new Multi-task Learning pipeline that trains simultaneously across multiple hate speech datasets.<n>We show strong results when examining the generalization error in train-test splits and substantial improvements when predicting on previously unseen datasets.
  arXiv  Detail & Related papers  (2022-08-22T21:13:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.