From Ground Trust to Truth: Disparities in Offensive Language Judgments on Contemporary Korean Political Discourse

• URL: http://arxiv.org/abs/2509.14712v1
• Date: Thu, 18 Sep 2025 07:57:18 GMT
• Title: From Ground Trust to Truth: Disparities in Offensive Language Judgments on Contemporary Korean Political Discourse
• Abstract summary: This study constructed a large-scale dataset of contemporary political discourse and employed three refined judgments in the absence of ground truth.<n>We identified distinct patterns for each judgment and demonstrated tendencies of label agreement using a leave-one-out strategy.<n>This suggests a feasible approach applicable in real-world settings with inherent constraints.
• Score: 19.382948880868522
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Although offensive language continually evolves over time, even recent studies using LLMs have predominantly relied on outdated datasets and rarely evaluated the generalization ability on unseen texts. In this study, we constructed a large-scale dataset of contemporary political discourse and employed three refined judgments in the absence of ground truth. Each judgment reflects a representative offensive language detection method and is carefully designed for optimal conditions. We identified distinct patterns for each judgment and demonstrated tendencies of label agreement using a leave-one-out strategy. By establishing pseudo-labels as ground trust for quantitative performance assessment, we observed that a strategically designed single prompting achieves comparable performance to more resource-intensive methods. This suggests a feasible approach applicable in real-world settings with inherent constraints.

Related papers

• VARM-Bench: Benchmarking Verifiable Structured Reasoning in Chinese Abusive Speech Moderation [12.885908330308046]
  VARM-Bench is a benchmark for deterministically verifying the stated basis of a moderation decision.<n>Our protocol evaluates field correctness, target alignment, output validity, complete-record agreement, and hidden record errors.<n>Results show that strong label-level performance can conceal substantial errors in complete moderation records.
  arXiv  Detail & Related papers  (2026-08-16T07:52:31Z)
• Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs [51.56484100374058]
  Large Language Models (LLMs) demonstrate remarkable multilingual fluency, but their internal knowledge representations remain disproportionately biased toward high-resource languages.<n>We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages.<n>Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-07-21T16:15:05Z)
• BIPOLAR: Polarization-based granular framework for LLM bias evaluation [0.0]
  This study proposes a reusable, granular, and topic-agnostic framework to evaluate polarisation-related biases in large language models.<n>Our approach combines polarisation-sensitive sentiment metrics with a synthetically generated balanced dataset of conflict-related statements.<n>As a case study, we created a synthetic dataset that focusses on the Russia-Ukraine war, and we evaluated the bias in several LLMs.
  arXiv  Detail & Related papers  (2025-08-14T20:44:19Z)
• Evaluating Style-Personalized Text Generation: Challenges and Directions [13.84471733325089]
  Style personalization is highly specific, relative to every user, and depends strongly on the pragmatic context.<n>We examine the effectiveness of the most common metrics used in the field, such as BLEU, embeddings, and LLMs-as-judges.<n>We find strong evidence that employing ensembles of diverse evaluation metrics consistently outperforms single-evaluator methods.
  arXiv  Detail & Related papers  (2025-08-08T15:07:31Z)
• Is LLM an Overconfident Judge? Unveiling the Capabilities of LLMs in Detecting Offensive Language with Annotation Disagreement [22.992484902761994]
  This study systematically evaluates the performance of multiple Large Language Models (LLMs) in detecting offensive language.<n>We analyze binary classification accuracy, examine the relationship between model confidence and human disagreement, and explore how disagreement samples influence model decision-making.
  arXiv  Detail & Related papers  (2025-02-10T07:14:26Z)
• Uncovering Biases with Reflective Large Language Models [2.5200794639628032]
  Biases and errors in human-labeled data present significant challenges for machine learning. We present the Reflective LLM Dialogue Framework RLDF, which leverages structured adversarial dialogues to uncover diverse perspectives. Experiments show RLDF successfully identifies potential biases in public content while exposing limitations in human-labeled data.
  arXiv  Detail & Related papers  (2024-08-24T04:48:32Z)
• Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective [66.34066553400108]
  We conduct a rigorous evaluation of large language models' implicit bias towards certain demographics.<n>Inspired by psychometric principles, we propose three attack approaches, i.e., Disguise, Deception, and Teaching.<n>Our methods can elicit LLMs' inner bias more effectively than competitive baselines.
  arXiv  Detail & Related papers  (2024-06-20T06:42:08Z)
• On the Robustness of Language Guidance for Low-Level Vision Tasks: Findings from Depth Estimation [71.72465617754553]
  We generate "low-level" sentences that convey object-centric, three-dimensional spatial relationships, incorporate them as additional language priors and evaluate their downstream impact on depth estimation. Our key finding is that current language-guided depth estimators perform optimally only with scene-level descriptions. Despite leveraging additional data, these methods are not robust to directed adversarial attacks and decline in performance with an increase in distribution shift.
  arXiv  Detail & Related papers  (2024-04-12T15:35:20Z)
• Exploring the Jungle of Bias: Political Bias Attribution in Language Models via Dependency Analysis [86.49858739347412]
  Large Language Models (LLMs) have sparked intense debate regarding the prevalence of bias in these models and its mitigation. We propose a prompt-based method for the extraction of confounding and mediating attributes which contribute to the decision process. We find that the observed disparate treatment can at least in part be attributed to confounding and mitigating attributes and model misalignment.
  arXiv  Detail & Related papers  (2023-11-15T00:02:25Z)
• Evaluating statistical language models as pragmatic reasoners [39.72348730045737]
  We evaluate the capacity of large language models to infer meanings of pragmatic utterances. We find that LLMs can derive context-grounded, human-like distributions over the interpretations of several complex pragmatic utterances. Results inform the inferential capacity of statistical language models, and their use in pragmatic and semantic parsing applications.
  arXiv  Detail & Related papers  (2023-05-01T18:22:10Z)
• We're Afraid Language Models Aren't Modeling Ambiguity [136.8068419824318]
  Managing ambiguity is a key part of human language understanding. We characterize ambiguity in a sentence by its effect on entailment relations with another sentence. We show that a multilabel NLI model can flag political claims in the wild that are misleading due to ambiguity.
  arXiv  Detail & Related papers  (2023-04-27T17:57:58Z)
• Learning-based Hybrid Local Search for the Hard-label Textual Attack [53.92227690452377]
  We consider a rarely investigated but more rigorous setting, namely hard-label attack, in which the attacker could only access the prediction label. Based on this observation, we propose a novel hard-label attack, called Learning-based Hybrid Local Search (LHLS) algorithm. Our LHLS significantly outperforms existing hard-label attacks regarding the attack performance as well as adversary quality.
  arXiv  Detail & Related papers  (2022-01-20T14:16:07Z)
• Semantics-Preserved Distortion for Personal Privacy Protection in Information Management [65.08939490413037]
  This paper suggests a linguistically-grounded approach to distort texts while maintaining semantic integrity. We present two distinct frameworks for semantic-preserving distortion: a generative approach and a substitutive approach. We also explore privacy protection in a specific medical information management scenario, showing our method effectively limits sensitive data memorization.
  arXiv  Detail & Related papers  (2022-01-04T04:01:05Z)
• Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models [86.02610674750345]
  Adversarial GLUE (AdvGLUE) is a new multi-task benchmark to explore and evaluate the vulnerabilities of modern large-scale language models under various types of adversarial attacks. We apply 14 adversarial attack methods to GLUE tasks to construct AdvGLUE, which is further validated by humans for reliable annotations. All the language models and robust training methods we tested perform poorly on AdvGLUE, with scores lagging far behind the benign accuracy.
  arXiv  Detail & Related papers  (2021-11-04T12:59:55Z)
• Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreement [7.288480094345606]
  We focus on the level of agreement among annotators while selecting data to create offensive language datasets. Our study comprises the creation of three novel datasets of English tweets covering different topics. We show that such hard cases, where low agreement is present, are not necessarily due to poor-quality annotation.
  arXiv  Detail & Related papers  (2021-09-28T08:55:04Z)
• Distributionally Robust Batch Contextual Bandits [20.667213458836734]
  Policy learning using historical observational data is an important problem that has found widespread applications. Existing literature rests on the crucial assumption that the future environment where the learned policy will be deployed is the same as the past environment. In this paper, we lift this assumption and aim to learn a distributionally robust policy with incomplete observational data.
  arXiv  Detail & Related papers  (2020-06-10T03:11:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.