Bridging Human and Model Perspectives: A Comparative Analysis of Political Bias Detection in News Media Using Large Language Models

• URL: http://arxiv.org/abs/2511.14606v1
• Date: Tue, 18 Nov 2025 15:58:04 GMT
• Title: Bridging Human and Model Perspectives: A Comparative Analysis of Political Bias Detection in News Media Using Large Language Models
• Authors: Shreya Adrita Banik, Niaz Nafi Rahman, Tahsina Moiukh, Farig Sadeque,
• Abstract summary: This study aims to present a comparative framework for evaluating the detection of political bias across human annotations and multiple language models.<n>We construct a manually annotated dataset of news articles and assess annotation consistency, bias polarity, and inter-model agreement.<n> Experimental results show that among traditional transformer-based models, RoBERTa achieves the highest alignment with human labels.
• Score: 0.3227658251731014
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Detecting political bias in news media is a complex task that requires interpreting subtle linguistic and contextual cues. Although recent advances in Natural Language Processing (NLP) have enabled automatic bias classification, the extent to which large language models (LLMs) align with human judgment still remains relatively underexplored and not yet well understood. This study aims to present a comparative framework for evaluating the detection of political bias across human annotations and multiple LLMs, including GPT, BERT, RoBERTa, and FLAN. We construct a manually annotated dataset of news articles and assess annotation consistency, bias polarity, and inter-model agreement to quantify divergence between human and model perceptions of bias. Experimental results show that among traditional transformer-based models, RoBERTa achieves the highest alignment with human labels, whereas generative models such as GPT demonstrate the strongest overall agreement with human annotations in a zero-shot setting. Among all transformer-based baselines, our fine-tuned RoBERTa model acquired the highest accuracy and the strongest alignment with human-annotated labels. Our findings highlight systematic differences in how humans and LLMs perceive political slant, underscoring the need for hybrid evaluation frameworks that combine human interpretability with model scalability in automated media bias detection.

Related papers

• Geopolitical Parallax: Beyond Walter Lippmann Just After Large Language Models [0.06372261626436676]
  This study investigates geopolitical parallax-systematic divergence in news quality and subjectivity assessments.<n>We compare article-level embeddings from Chinese-origin (Qwen, BGE, Jina) and Western-origin (Snowflake, Granite) model families.<n>Our findings reveal consistent, non-random divergences aligned with model origin.
  arXiv  Detail & Related papers  (2025-08-27T00:39:59Z)
• BIPOLAR: Polarization-based granular framework for LLM bias evaluation [0.0]
  This study proposes a reusable, granular, and topic-agnostic framework to evaluate polarisation-related biases in large language models.<n>Our approach combines polarisation-sensitive sentiment metrics with a synthetically generated balanced dataset of conflict-related statements.<n>As a case study, we created a synthetic dataset that focusses on the Russia-Ukraine war, and we evaluated the bias in several LLMs.
  arXiv  Detail & Related papers  (2025-08-14T20:44:19Z)
• Modeling Open-World Cognition as On-Demand Synthesis of Probabilistic Models [93.1043186636177]
  We explore the hypothesis that people use a combination of distributed and symbolic representations to construct bespoke mental models tailored to novel situations.<n>We propose a computational implementation of this idea -- a Model Synthesis Architecture''<n>We evaluate our MSA as a model of human judgments on a novel reasoning dataset.
  arXiv  Detail & Related papers  (2025-07-16T18:01:03Z)
• To Bias or Not to Bias: Detecting bias in News with bias-detector [1.8024397171920885]
  We perform sentence-level bias classification by fine-tuning a RoBERTa-based model on the expert-annotated BABE dataset.<n>We show statistically significant improvements in performance when comparing our model to a domain-adaptively pre-trained DA-RoBERTa baseline.<n>Our findings contribute to building more robust, explainable, and socially responsible NLP systems for media bias detection.
  arXiv  Detail & Related papers  (2025-05-19T11:54:39Z)
• Fine-Grained Bias Detection in LLM: Enhancing detection mechanisms for nuanced biases [0.0]
  This study presents a detection framework to identify nuanced biases in Large Language Models (LLMs)<n>The approach integrates contextual analysis, interpretability via attention mechanisms, and counterfactual data augmentation to capture hidden biases.<n>Results show improvements in detecting subtle biases compared to conventional methods.
  arXiv  Detail & Related papers  (2025-03-08T04:43:01Z)
• Identifying and Mitigating Social Bias Knowledge in Language Models [52.52955281662332]
  We propose a novel debiasing approach, Fairness Stamp (FAST), which enables fine-grained calibration of individual social biases.<n>FAST surpasses state-of-the-art baselines with superior debiasing performance.<n>This highlights the potential of fine-grained debiasing strategies to achieve fairness in large language models.
  arXiv  Detail & Related papers  (2024-08-07T17:14:58Z)
• Bias in Language Models: Beyond Trick Tests and Toward RUTEd Evaluation [49.3814117521631]
  Standard benchmarks of bias and fairness in large language models (LLMs) measure the association between the user attributes stated or implied by a prompt.<n>We develop analogous RUTEd evaluations from three contexts of real-world use: children's bedtime stories, user personas, and English language learning exercises.<n>We find that standard bias metrics have no significant correlation with the more realistic bias metrics.
  arXiv  Detail & Related papers  (2024-02-20T01:49:15Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)
• CBBQ: A Chinese Bias Benchmark Dataset Curated with Human-AI Collaboration for Large Language Models [52.25049362267279]
  We present a Chinese Bias Benchmark dataset that consists of over 100K questions jointly constructed by human experts and generative language models. The testing instances in the dataset are automatically derived from 3K+ high-quality templates manually authored with stringent quality control. Extensive experiments demonstrate the effectiveness of the dataset in detecting model bias, with all 10 publicly available Chinese large language models exhibiting strong bias in certain categories.
  arXiv  Detail & Related papers  (2023-06-28T14:14:44Z)
• Balancing out Bias: Achieving Fairness Through Training Reweighting [58.201275105195485]
  Bias in natural language processing arises from models learning characteristics of the author such as gender and race. Existing methods for mitigating and measuring bias do not directly account for correlations between author demographics and linguistic variables. This paper introduces a very simple but highly effective method for countering bias using instance reweighting.
  arXiv  Detail & Related papers  (2021-09-16T23:40:28Z)
• Inflating Topic Relevance with Ideology: A Case Study of Political Ideology Bias in Social Topic Detection Models [16.279854003220418]
  We investigate the impact of political ideology biases in training data. Our work highlights the susceptibility of large, complex models to propagating the biases from human-selected input. As a way to mitigate the bias, we propose to learn a text representation that is invariant to political ideology while still judging topic relevance.
  arXiv  Detail & Related papers  (2020-11-29T05:54:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.