Introducing MBIB -- the first Media Bias Identification Benchmark Task and Dataset Collection

• URL: http://arxiv.org/abs/2304.13148v1
• Date: Tue, 25 Apr 2023 20:49:55 GMT
• Title: Introducing MBIB -- the first Media Bias Identification Benchmark Task and Dataset Collection
• Authors: Martin Wessel, Tom\'a\v{s} Horych, Terry Ruas, Akiko Aizawa, Bela Gipp and Timo Spinde
• Abstract summary: We introduce the Media Bias Identification Benchmark (MBIB) to group different types of media bias under a common framework. After reviewing 115 datasets, we select nine tasks and carefully propose 22 associated datasets for evaluating media bias detection techniques. Our results suggest that while hate speech, racial bias, and gender bias are easier to detect, models struggle to handle certain bias types, e.g., cognitive and political bias.
• Score: 24.35462897801079
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Although media bias detection is a complex multi-task problem, there is, to date, no unified benchmark grouping these evaluation tasks. We introduce the Media Bias Identification Benchmark (MBIB), a comprehensive benchmark that groups different types of media bias (e.g., linguistic, cognitive, political) under a common framework to test how prospective detection techniques generalize. After reviewing 115 datasets, we select nine tasks and carefully propose 22 associated datasets for evaluating media bias detection techniques. We evaluate MBIB using state-of-the-art Transformer techniques (e.g., T5, BART). Our results suggest that while hate speech, racial bias, and gender bias are easier to detect, models struggle to handle certain bias types, e.g., cognitive and political bias. However, our results show that no single technique can outperform all the others significantly. We also find an uneven distribution of research interest and resource allocation to the individual tasks in media bias. A unified benchmark encourages the development of more robust systems and shifts the current paradigm in media bias detection evaluation towards solutions that tackle not one but multiple media bias types simultaneously.

Related papers

• VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Model [72.13121434085116]
  VLBiasBench is a benchmark aimed at evaluating biases in Large Vision-Language Models (LVLMs) We construct a dataset encompassing nine distinct categories of social biases, including age, disability status, gender, nationality, physical appearance, race, religion, profession, social economic status and two intersectional bias categories (race x gender, and race x social economic status) We conduct extensive evaluations on 15 open-source models as well as one advanced closed-source model, providing some new insights into the biases revealing from these models.
  arXiv  Detail & Related papers  (2024-06-20T10:56:59Z)
• Take Care of Your Prompt Bias! Investigating and Mitigating Prompt Bias in Factual Knowledge Extraction [56.17020601803071]
  Recent research shows that pre-trained language models (PLMs) suffer from "prompt bias" in factual knowledge extraction. This paper aims to improve the reliability of existing benchmarks by thoroughly investigating and mitigating prompt bias.
  arXiv  Detail & Related papers  (2024-03-15T02:04:35Z)
• The Media Bias Taxonomy: A Systematic Literature Review on the Forms and Automated Detection of Media Bias [5.579028648465784]
  This article summarizes the research on computational methods to detect media bias by systematically reviewing 3140 research papers published between 2019 and 2022. We show that media bias detection is a highly active research field, in which transformer-based classification approaches have led to significant improvements in recent years.
  arXiv  Detail & Related papers  (2023-12-26T18:13:52Z)
• Mitigating Bias for Question Answering Models by Tracking Bias Influence [84.66462028537475]
  We propose BMBI, an approach to mitigate the bias of multiple-choice QA models. Based on the intuition that a model would lean to be more biased if it learns from a biased example, we measure the bias level of a query instance. We show that our method could be applied to multiple QA formulations across multiple bias categories.
  arXiv  Detail & Related papers  (2023-10-13T00:49:09Z)
• Target-Aware Contextual Political Bias Detection in News [22.396285428304083]
  Sentence-level political bias detection in news is a challenging task that requires an understanding of bias in consideration of the context. Previous work in media bias detection has proposed augmentation techniques to exploit this fact. We propose techniques to more carefully search for context using a bias-sensitive, target-aware approach for data augmentation.
  arXiv  Detail & Related papers  (2023-10-02T12:25:05Z)
• Exploiting Transformer-based Multitask Learning for the Detection of Media Bias in News Articles [21.960154864540282]
  We propose a Transformer-based deep learning architecture trained via Multi-Task Learning to detect media bias. Our best-performing implementation achieves a macro $F_1$ of 0.776, a performance boost of 3% compared to our baseline, outperforming existing methods.
  arXiv  Detail & Related papers  (2022-11-07T12:22:31Z)
• Neural Media Bias Detection Using Distant Supervision With BABE -- Bias Annotations By Experts [24.51774048437496]
  This paper presents BABE, a robust and diverse data set for media bias research. It consists of 3,700 sentences balanced among topics and outlets, containing media bias labels on the word and sentence level. Based on our data, we also introduce a way to detect bias-inducing sentences in news articles automatically.
  arXiv  Detail & Related papers  (2022-09-29T05:32:55Z)
• Mitigating Representation Bias in Action Recognition: Algorithms and Benchmarks [76.35271072704384]
  Deep learning models perform poorly when applied to videos with rare scenes or objects. We tackle this problem from two different angles: algorithm and dataset. We show that the debiased representation can generalize better when transferred to other datasets and tasks.
  arXiv  Detail & Related papers  (2022-09-20T00:30:35Z)
• A Domain-adaptive Pre-training Approach for Language Bias Detection in News [3.7238620986236373]
  We present DA-RoBERTa, a new state-of-the-art transformer-based model adapted to the media bias domain. We also train, DA-BERT and DA-BART, two more transformer models adapted to the bias domain. Our proposed domain-adapted models outperform prior bias detection approaches on the same data.
  arXiv  Detail & Related papers  (2022-05-22T08:18:19Z)
• Unbiased Math Word Problems Benchmark for Mitigating Solving Bias [72.8677805114825]
  Current solvers exist solving bias which consists of data bias and learning bias due to biased dataset and improper training strategy. Our experiments verify MWP solvers are easy to be biased by the biased training datasets which do not cover diverse questions for each problem narrative of all MWPs. An MWP can be naturally solved by multiple equivalent equations while current datasets take only one of the equivalent equations as ground truth.
  arXiv  Detail & Related papers  (2022-05-17T06:07:04Z)
• Information-Theoretic Bias Reduction via Causal View of Spurious Correlation [71.9123886505321]
  We propose an information-theoretic bias measurement technique through a causal interpretation of spurious correlation. We present a novel debiasing framework against the algorithmic bias, which incorporates a bias regularization loss. The proposed bias measurement and debiasing approaches are validated in diverse realistic scenarios.
  arXiv  Detail & Related papers  (2022-01-10T01:19:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.