When Bigger Isn't Better: A Comprehensive Fairness Evaluation of Political Bias in Multi-News Summarisation

• URL: http://arxiv.org/abs/2604.21309v1
• Date: Thu, 23 Apr 2026 06:03:24 GMT
• Title: When Bigger Isn't Better: A Comprehensive Fairness Evaluation of Political Bias in Multi-News Summarisation
• Abstract summary: Multi-document news summarisation systems can exhibit political bias through unequal representation of viewpoints, disproportionate emphasis on certain perspectives, and systematic underrepresentation of minority voices.<n>This study presents a comprehensive evaluation of such bias in multi-document news summarisation using FairNews, a dataset of complete news articles with political orientation labels.
• Score: 29.441760068089508
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-document news summarisation systems are increasingly adopted for their convenience in processing vast daily news content, making fairness across diverse political perspectives critical. However, these systems can exhibit political bias through unequal representation of viewpoints, disproportionate emphasis on certain perspectives, and systematic underrepresentation of minority voices. This study presents a comprehensive evaluation of such bias in multi-document news summarisation using FairNews, a dataset of complete news articles with political orientation labels, examining how large language models (LLMs) handle sources with varying political leanings across 13 models and five fairness metrics. We investigate both baseline model performance and effectiveness of various debiasing interventions, including prompt-based and judge-based approaches. Our findings challenge the assumption that larger models yield fairer outputs, as mid-sized variants consistently outperform their larger counterparts, offering the best balance of fairness and efficiency. Prompt-based debiasing proves highly model dependent, while entity sentiment emerges as the most stubborn fairness dimension, resisting all intervention strategies tested. These results demonstrate that fairness in multi-document news summarisation requires multi-dimensional evaluation frameworks and targeted, architecture-aware debiasing rather than simply scaling up.

Related papers

• HoloFair: Unified T2I Fairness Evaluation and Fair-GRPO Debiasing [20.795940054868854]
  Existing evaluation methods address only single-dimensional biases.<n>We introduce HoloFair, a comprehensive benchmark framework for multidimensional demographic bias analysis.<n>We also introduce Fair-GRPO, a reinforcement-learning-based debiasing method.
  arXiv  Detail & Related papers  (2026-05-23T17:52:07Z)
• A Multi-Dimensional Audit of Politically Aligned Large Language Models [3.147897243459717]
  This work aims to ensure politically-aligned language models generate legitimate, harmless arguments.<n>We propose a multi-dimensional framework inspired by Habermas' Theory of Communicative Action to audit politically aligned language models.
  arXiv  Detail & Related papers  (2026-04-27T12:57:52Z)
• The Unfairness of Multifactorial Bias in Recommendation [68.35079031029616]
  Popularity bias and positivity bias are prominent sources of bias in recommender systems.<n>In this work, we examine how multifactorial bias influences item-side fairness.<n>We adapt a percentile-based rating transformation as a pre-processing strategy to mitigate multifactorial bias.
  arXiv  Detail & Related papers  (2026-01-19T08:37:43Z)
• A Systematic Analysis of Biases in Large Language Models [40.23320093091831]
  Large language models (LLMs) have rapidly become indispensable tools for acquiring information and supporting human decision-making.<n>This study examines their underlying biases and inclinations across the dimensions of politics, ideology, alliance, language, and gender.<n>Results indicate that while the LLMs are aligned to be neutral and impartial, they still show biases and affinities of different types.
  arXiv  Detail & Related papers  (2025-12-16T03:38:08Z)
• Unraveling Media Perspectives: A Comprehensive Methodology Combining Large Language Models, Topic Modeling, Sentiment Analysis, and Ontology Learning to Analyse Media Bias [0.0]
  This study introduces a novel methodology for scalable, minimally biased analysis of media bias in political news.<n>The proposed approach examines event selection, labeling, word choice, and commission and omission biases across news sources.
  arXiv  Detail & Related papers  (2025-05-03T09:09:34Z)
• Mapping the Media Landscape: Predicting Factual Reporting and Political Bias Through Web Interactions [0.7249731529275342]
  We propose an extension to a recently presented news media reliability estimation method. We assess the classification performance of four reinforcement learning strategies on a large news media hyperlink graph. Our experiments, targeting two challenging bias descriptors, factual reporting and political bias, showed a significant performance improvement at the source media level.
  arXiv  Detail & Related papers  (2024-10-23T08:18:26Z)
• Understanding Position Bias Effects on Fairness in Social Multi-Document Summarization [1.9950682531209158]
  We investigate the effect of group ordering in input documents when summarizing tweets from three linguistic communities. Our results suggest that position bias manifests differently in social multi-document summarization.
  arXiv  Detail & Related papers  (2024-05-03T00:19:31Z)
• GPTBIAS: A Comprehensive Framework for Evaluating Bias in Large Language Models [83.30078426829627]
  Large language models (LLMs) have gained popularity and are being widely adopted by a large user community. The existing evaluation methods have many constraints, and their results exhibit a limited degree of interpretability. We propose a bias evaluation framework named GPTBIAS that leverages the high performance of LLMs to assess bias in models.
  arXiv  Detail & Related papers  (2023-12-11T12:02:14Z)
• Fair Abstractive Summarization of Diverse Perspectives [103.08300574459783]
  A fair summary should provide a comprehensive coverage of diverse perspectives without underrepresenting certain groups. We first formally define fairness in abstractive summarization as not underrepresenting perspectives of any groups of people. We propose four reference-free automatic metrics by measuring the differences between target and source perspectives.
  arXiv  Detail & Related papers  (2023-11-14T03:38:55Z)
• Fair Enough: Standardizing Evaluation and Model Selection for Fairness Research in NLP [64.45845091719002]
  Modern NLP systems exhibit a range of biases, which a growing literature on model debiasing attempts to correct. This paper seeks to clarify the current situation and plot a course for meaningful progress in fair learning.
  arXiv  Detail & Related papers  (2023-02-11T14:54:00Z)
• Unveiling the Hidden Agenda: Biases in News Reporting and Consumption [59.55900146668931]
  We build a six-year dataset on the Italian vaccine debate and adopt a Bayesian latent space model to identify narrative and selection biases. We found a nonlinear relationship between biases and engagement, with higher engagement for extreme positions. Analysis of news consumption on Twitter reveals common audiences among news outlets with similar ideological positions.
  arXiv  Detail & Related papers  (2023-01-14T18:58:42Z)
• NeuS: Neutral Multi-News Summarization for Mitigating Framing Bias [54.89737992911079]
  We propose a new task, a neutral summary generation from multiple news headlines of the varying political spectrum. One of the most interesting observations is that generation models can hallucinate not only factually inaccurate or unverifiable content, but also politically biased content.
  arXiv  Detail & Related papers  (2022-04-11T07:06:01Z)
• Enabling News Consumers to View and Understand Biased News Coverage: A Study on the Perception and Visualization of Media Bias [7.092487352312782]
  We create three manually annotated datasets and test varying visualization strategies. Results show no strong effects of becoming aware of the bias of the treatment groups compared to the control group. Using a multilevel model, we find that perceived journalist bias is significantly related to perceived political extremeness and impartiality of the article.
  arXiv  Detail & Related papers  (2021-05-20T10:16:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.