Understanding Position Bias Effects on Fairness in Social Multi-Document Summarization

• URL: http://arxiv.org/abs/2405.01790v1
• Date: Fri, 3 May 2024 00:19:31 GMT
• Title: Understanding Position Bias Effects on Fairness in Social Multi-Document Summarization
• Authors: Olubusayo Olabisi, Ameeta Agrawal,
• Abstract summary: We investigate the effect of group ordering in input documents when summarizing tweets from three linguistic communities. Our results suggest that position bias manifests differently in social multi-document summarization.
• Score: 1.9950682531209158
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text summarization models have typically focused on optimizing aspects of quality such as fluency, relevance, and coherence, particularly in the context of news articles. However, summarization models are increasingly being used to summarize diverse sources of text, such as social media data, that encompass a wide demographic user base. It is thus crucial to assess not only the quality of the generated summaries, but also the extent to which they can fairly represent the opinions of diverse social groups. Position bias, a long-known issue in news summarization, has received limited attention in the context of social multi-document summarization. We deeply investigate this phenomenon by analyzing the effect of group ordering in input documents when summarizing tweets from three distinct linguistic communities: African-American English, Hispanic-aligned Language, and White-aligned Language. Our empirical analysis shows that although the textual quality of the summaries remains consistent regardless of the input document order, in terms of fairness, the results vary significantly depending on how the dialect groups are presented in the input data. Our results suggest that position bias manifests differently in social multi-document summarization, severely impacting the fairness of summarization models.

Related papers

• Fair Abstractive Summarization of Diverse Perspectives [103.08300574459783]
  A fair summary should provide a comprehensive coverage of diverse perspectives without underrepresenting certain groups. We first formally define fairness in abstractive summarization as not underrepresenting perspectives of any groups of people. We propose four reference-free automatic metrics by measuring the differences between target and source perspectives.
  arXiv  Detail & Related papers  (2023-11-14T03:38:55Z)
• Comprehending Lexical and Affective Ontologies in the Demographically Diverse Spatial Social Media Discourse [0.0]
  This study aims to comprehend linguistic and socio-demographic features, encompassing English language styles, conveyed sentiments, and lexical diversity within social media data. Our analysis entails the extraction and examination of various statistical, grammatical, and sentimental features from two groups. Our investigation unveils substantial disparities in certain linguistic attributes between the two groups, yielding a macro F1 score of approximately 0.85.
  arXiv  Detail & Related papers  (2023-11-12T04:23:33Z)
• On Context Utilization in Summarization with Large Language Models [83.84459732796302]
  Large language models (LLMs) excel in abstractive summarization tasks, delivering fluent and pertinent summaries. Recent advancements have extended their capabilities to handle long-input contexts, exceeding 100k tokens. We conduct the first comprehensive study on context utilization and position bias in summarization.
  arXiv  Detail & Related papers  (2023-10-16T16:45:12Z)
• Bias in News Summarization: Measures, Pitfalls and Corpora [4.917075909999548]
  We introduce definitions for biased behaviours in summarization models, along with practical operationalizations. We measure gender bias in English summaries generated by both purpose-built summarization models and general purpose chat models. We find content selection in single document summarization to be largely unaffected by gender bias, while hallucinations exhibit evidence of bias.
  arXiv  Detail & Related papers  (2023-09-14T22:20:27Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)
• Large Language Models are Diverse Role-Players for Summarization Evaluation [82.31575622685902]
  A document summary's quality can be assessed by human annotators on various criteria, both objective ones like grammar and correctness, and subjective ones like informativeness, succinctness, and appeal. Most of the automatic evaluation methods like BLUE/ROUGE may be not able to adequately capture the above dimensions. We propose a new evaluation framework based on LLMs, which provides a comprehensive evaluation framework by comparing generated text and reference text from both objective and subjective aspects.
  arXiv  Detail & Related papers  (2023-03-27T10:40:59Z)
• On the State of German (Abstractive) Text Summarization [3.1776833268555134]
  We assess the landscape of German abstractive text summarization. We investigate why practically useful solutions for abstractive text summarization are still absent in industry.
  arXiv  Detail & Related papers  (2023-01-17T18:59:20Z)
• Modeling Multi-level Context for Informational Bias Detection by Contrastive Learning and Sentential Graph Network [13.905580921329717]
  Informational bias can only be detected together with the context. In this paper, we integrate three levels of context to detect the sentence-level informational bias in English news articles. Our model, MultiCTX, uses contrastive learning and sentence graphs together with Graph Attention Network (GAT) to encode these three degrees of context.
  arXiv  Detail & Related papers  (2022-01-25T15:07:09Z)
• Towards Understanding and Mitigating Social Biases in Language Models [107.82654101403264]
  Large-scale pretrained language models (LMs) can be potentially dangerous in manifesting undesirable representational biases. We propose steps towards mitigating social biases during text generation. Our empirical results and human evaluation demonstrate effectiveness in mitigating bias while retaining crucial contextual information.
  arXiv  Detail & Related papers  (2021-06-24T17:52:43Z)
• Fairness for Whom? Understanding the Reader's Perception of Fairness in Text Summarization [9.136419921943235]
  We study the interplay between the fairness notions and how readers perceive them in textual summaries. Standard ROUGE evaluation metrics are unable to quantify the perceived (un)fairness of the summaries.
  arXiv  Detail & Related papers  (2021-01-29T05:14:34Z)
• Weakly-Supervised Aspect-Based Sentiment Analysis via Joint Aspect-Sentiment Topic Embedding [71.2260967797055]
  We propose a weakly-supervised approach for aspect-based sentiment analysis. We learn sentiment, aspect> joint topic embeddings in the word embedding space. We then use neural models to generalize the word-level discriminative information.
  arXiv  Detail & Related papers  (2020-10-13T21:33:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.