McBE: A Multi-task Chinese Bias Evaluation Benchmark for Large Language Models

• URL: http://arxiv.org/abs/2507.02088v2
• Date: Thu, 07 Aug 2025 11:09:06 GMT
• Title: McBE: A Multi-task Chinese Bias Evaluation Benchmark for Large Language Models
• Authors: Tian Lan, Xiangdong Su, Xu Liu, Ruirui Wang, Ke Chang, Jiang Li, Guanglai Gao,
• Abstract summary: We present a Multi-task Chinese Bias Evaluation Benchmark (McBE) that includes 4,077 bias evaluation instances.<n>This dataset provides extensive category coverage, content diversity, and measuring comprehensiveness.<n>We conduct an in-depth analysis of results, offering novel insights into bias in large language models (LLMs)
• Score: 26.202296897643382
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As large language models (LLMs) are increasingly applied to various NLP tasks, their inherent biases are gradually disclosed. Therefore, measuring biases in LLMs is crucial to mitigate its ethical risks. However, most existing bias evaluation datasets focus on English and North American culture, and their bias categories are not fully applicable to other cultures. The datasets grounded in the Chinese language and culture are scarce. More importantly, these datasets usually only support single evaluation tasks and cannot evaluate the bias from multiple aspects in LLMs. To address these issues, we present a Multi-task Chinese Bias Evaluation Benchmark (McBE) that includes 4,077 bias evaluation instances, covering 12 single bias categories, 82 subcategories and introducing 5 evaluation tasks, providing extensive category coverage, content diversity, and measuring comprehensiveness. Additionally, we evaluate several popular LLMs from different series and with parameter sizes. In general, all these LLMs demonstrated varying degrees of bias. We conduct an in-depth analysis of results, offering novel insights into bias in LLMs.

Related papers

• Evaluating how LLM annotations represent diverse views on contentious topics [3.405231040967506]
  We show that generative large language models (LLMs) tend to be biased in the same directions on the same demographic categories within the same datasets.<n>We conclude with a discussion of the implications for researchers and practitioners using LLMs for automated data annotation tasks.
  arXiv  Detail & Related papers  (2025-03-29T22:53:15Z)
• No LLM is Free From Bias: A Comprehensive Study of Bias Evaluation in Large Language Models [0.9620910657090186]
  Large Language Models (LLMs) have increased the performance of different natural language understanding as well as generation tasks.<n>We provide a unified evaluation of benchmarks using a set of representative small and medium-sized LLMs.<n>We propose five prompting approaches to carry out the bias detection task across different aspects of bias.<n>The results indicate that each of the selected LLMs suffer from one or the other form of bias with the Phi-3.5B model being the least biased.
  arXiv  Detail & Related papers  (2025-03-15T03:58:14Z)
• Implicit Bias in LLMs: A Survey [2.07180164747172]
  This paper provides a comprehensive review of the existing literature on implicit bias in Large language models.<n>We begin by introducing key concepts, theories and methods related to implicit bias in psychology.<n>We categorize detection methods into three primary approaches: word association, task-oriented text generation and decision-making.
  arXiv  Detail & Related papers  (2025-03-04T16:49:37Z)
• Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge [84.34545223897578]
  Despite their excellence in many domains, potential issues are under-explored, undermining their reliability and the scope of their utility. We identify 12 key potential biases and propose a new automated bias quantification framework-CALM- which quantifies and analyzes each type of bias in LLM-as-a-Judge. Our work highlights the need for stakeholders to address these issues and remind users to exercise caution in LLM-as-a-Judge applications.
  arXiv  Detail & Related papers  (2024-10-03T17:53:30Z)
• CEB: Compositional Evaluation Benchmark for Fairness in Large Language Models [58.57987316300529]
  Large Language Models (LLMs) are increasingly deployed to handle various natural language processing (NLP) tasks.<n>To evaluate the biases exhibited by LLMs, researchers have recently proposed a variety of datasets.<n>We propose CEB, a Compositional Evaluation Benchmark that covers different types of bias across different social groups and tasks.
  arXiv  Detail & Related papers  (2024-07-02T16:31:37Z)
• Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective [66.34066553400108]
  We conduct a rigorous evaluation of large language models' implicit bias towards certain demographics.<n>Inspired by psychometric principles, we propose three attack approaches, i.e., Disguise, Deception, and Teaching.<n>Our methods can elicit LLMs' inner bias more effectively than competitive baselines.
  arXiv  Detail & Related papers  (2024-06-20T06:42:08Z)
• CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models [53.9835961434552]
  We introduce the Chinese Instruction-Following Benchmark (CIF-Bench) to evaluate the generalizability of large language models (LLMs) to the Chinese language. CIF-Bench comprises 150 tasks and 15,000 input-output pairs, developed by native speakers to test complex reasoning and Chinese cultural nuances. To mitigate data contamination, we release only half of the dataset publicly, with the remainder kept private, and introduce diversified instructions to minimize score variance.
  arXiv  Detail & Related papers  (2024-02-20T16:02:12Z)
• Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement [75.7148545929689]
  Large language models (LLMs) improve their performance through self-feedback on certain tasks while degrade on others. We formally define LLM's self-bias - the tendency to favor its own generation. We analyze six LLMs on translation, constrained text generation, and mathematical reasoning tasks.
  arXiv  Detail & Related papers  (2024-02-18T03:10:39Z)
• GPTBIAS: A Comprehensive Framework for Evaluating Bias in Large Language Models [83.30078426829627]
  Large language models (LLMs) have gained popularity and are being widely adopted by a large user community. The existing evaluation methods have many constraints, and their results exhibit a limited degree of interpretability. We propose a bias evaluation framework named GPTBIAS that leverages the high performance of LLMs to assess bias in models.
  arXiv  Detail & Related papers  (2023-12-11T12:02:14Z)
• Investigating Subtler Biases in LLMs: Ageism, Beauty, Institutional, and Nationality Bias in Generative Models [0.0]
  This paper investigates bias along less-studied but still consequential, dimensions, such as age and beauty. We ask whether LLMs hold wide-reaching biases of positive or negative sentiment for specific social groups similar to the "what is beautiful is good" bias found in people in experimental psychology.
  arXiv  Detail & Related papers  (2023-09-16T07:07:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.