BiasLab: Toward Explainable Political Bias Detection with Dual-Axis Annotations and Rationale Indicators

• URL: http://arxiv.org/abs/2505.16081v1
• Date: Wed, 21 May 2025 23:50:42 GMT
• Title: BiasLab: Toward Explainable Political Bias Detection with Dual-Axis Annotations and Rationale Indicators
• Authors: KMA Solaiman,
• Abstract summary: BiasLab is a dataset of 300 political news articles annotated for perceived ideological bias.<n>Each article is labeled by crowdworkers along two independent scales, assessing sentiment toward the Democratic and Republican parties.<n>We quantify inter-annotator agreement, analyze misalignment with source-level outlet bias, and organize the resulting labels into interpretable subsets.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We present BiasLab, a dataset of 300 political news articles annotated for perceived ideological bias. These articles were selected from a curated 900-document pool covering diverse political events and source biases. Each article is labeled by crowdworkers along two independent scales, assessing sentiment toward the Democratic and Republican parties, and enriched with rationale indicators. The annotation pipeline incorporates targeted worker qualification and was refined through pilot-phase analysis. We quantify inter-annotator agreement, analyze misalignment with source-level outlet bias, and organize the resulting labels into interpretable subsets. Additionally, we simulate annotation using schema-constrained GPT-4o, enabling direct comparison to human labels and revealing mirrored asymmetries, especially in misclassifying subtly right-leaning content. We define two modeling tasks: perception drift prediction and rationale type classification, and report baseline performance to illustrate the challenge of explainable bias detection. BiasLab's rich rationale annotations provide actionable interpretations that facilitate explainable modeling of political bias, supporting the development of transparent, socially aware NLP systems. We release the dataset, annotation schema, and modeling code to encourage research on human-in-the-loop interpretability and the evaluation of explanation effectiveness in real-world settings.

Related papers

• PRISM: A Framework for Producing Interpretable Political Bias Embeddings with Political-Aware Cross-Encoder [17.00358234728804]
  PRISM is a framework to produce inteRpretable polItical biaS eMbeddings.<n>It extracts political topics and their corresponding bias indicators from weakly labeled news data.<n>It then assigns structured bias scores to news articles based on their alignment with these indicators.
  arXiv  Detail & Related papers  (2025-05-30T14:31:53Z)
• Evaluate Bias without Manual Test Sets: A Concept Representation Perspective for LLMs [25.62533031580287]
  Bias in Large Language Models (LLMs) significantly undermines their reliability and fairness.<n>We propose BiasLens, a test-set-free bias analysis framework based on the structure of the model's vector space.
  arXiv  Detail & Related papers  (2025-05-21T13:50:23Z)
• COBIAS: Assessing the Contextual Reliability of Bias Benchmarks for Language Models [14.594920595573038]
  Large Language Models (LLMs) often inherit biases from the web data they are trained on, which contains stereotypes and prejudices.<n>Current methods for evaluating and mitigating these biases rely on bias-benchmark datasets.<n>We introduce a contextual reliability framework, which evaluates model robustness to biased statements by considering the various contexts in which they may appear.
  arXiv  Detail & Related papers  (2024-02-22T10:46:11Z)
• Fair Abstractive Summarization of Diverse Perspectives [103.08300574459783]
  A fair summary should provide a comprehensive coverage of diverse perspectives without underrepresenting certain groups. We first formally define fairness in abstractive summarization as not underrepresenting perspectives of any groups of people. We propose four reference-free automatic metrics by measuring the differences between target and source perspectives.
  arXiv  Detail & Related papers  (2023-11-14T03:38:55Z)
• Evaluating the Fairness of Discriminative Foundation Models in Computer Vision [51.176061115977774]
  We propose a novel taxonomy for bias evaluation of discriminative foundation models, such as Contrastive Language-Pretraining (CLIP) We then systematically evaluate existing methods for mitigating bias in these models with respect to our taxonomy. Specifically, we evaluate OpenAI's CLIP and OpenCLIP models for key applications, such as zero-shot classification, image retrieval and image captioning.
  arXiv  Detail & Related papers  (2023-10-18T10:32:39Z)
• Bias in News Summarization: Measures, Pitfalls and Corpora [4.917075909999548]
  We introduce definitions for biased behaviours in summarization models, along with practical operationalizations. We measure gender bias in English summaries generated by both purpose-built summarization models and general purpose chat models. We find content selection in single document summarization to be largely unaffected by gender bias, while hallucinations exhibit evidence of bias.
  arXiv  Detail & Related papers  (2023-09-14T22:20:27Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)
• Metrics for Dataset Demographic Bias: A Case Study on Facial Expression Recognition [4.336779198334903]
  One of the most prominent types of demographic bias are statistical imbalances in the representation of demographic groups in the datasets. We develop a taxonomy for the classification of these metrics, providing a practical guide for the selection of appropriate metrics. The paper provides valuable insights for researchers in AI and related fields to mitigate dataset bias and improve the fairness and accuracy of AI models.
  arXiv  Detail & Related papers  (2023-03-28T11:04:18Z)
• Filtering Context Mitigates Scarcity and Selection Bias in Political Ideology Prediction [42.31457743674423]
  We propose a novel supervised learning approach for political ideology prediction (PIP) We show that our model is capable of outputting predictions even when trained with as little as 5% biased data.
  arXiv  Detail & Related papers  (2023-02-01T04:34:48Z)
• D-BIAS: A Causality-Based Human-in-the-Loop System for Tackling Algorithmic Bias [57.87117733071416]
  We propose D-BIAS, a visual interactive tool that embodies human-in-the-loop AI approach for auditing and mitigating social biases. A user can detect the presence of bias against a group by identifying unfair causal relationships in the causal network. For each interaction, say weakening/deleting a biased causal edge, the system uses a novel method to simulate a new (debiased) dataset.
  arXiv  Detail & Related papers  (2022-08-10T03:41:48Z)
• NeuS: Neutral Multi-News Summarization for Mitigating Framing Bias [54.89737992911079]
  We propose a new task, a neutral summary generation from multiple news headlines of the varying political spectrum. One of the most interesting observations is that generation models can hallucinate not only factually inaccurate or unverifiable content, but also politically biased content.
  arXiv  Detail & Related papers  (2022-04-11T07:06:01Z)
• Balancing out Bias: Achieving Fairness Through Training Reweighting [58.201275105195485]
  Bias in natural language processing arises from models learning characteristics of the author such as gender and race. Existing methods for mitigating and measuring bias do not directly account for correlations between author demographics and linguistic variables. This paper introduces a very simple but highly effective method for countering bias using instance reweighting.
  arXiv  Detail & Related papers  (2021-09-16T23:40:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.