Assessing Judging Bias in Large Reasoning Models: An Empirical Study

• URL: http://arxiv.org/abs/2504.09946v2
• Date: Fri, 18 Apr 2025 02:05:38 GMT
• Title: Assessing Judging Bias in Large Reasoning Models: An Empirical Study
• Authors: Qian Wang, Zhanzhi Lou, Zhenheng Tang, Nuo Chen, Xuandong Zhao, Wenxuan Zhang, Dawn Song, Bingsheng He,
• Abstract summary: Large Reasoning Models (LRMs) like DeepSeek-R1 and OpenAI-o1 have demonstrated remarkable reasoning capabilities.<n>We present a benchmark comparing judging biases between LLMs and LRMs across both subjective preference-alignment datasets and objective fact-based datasets.
• Score: 99.86300466350013
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Reasoning Models (LRMs) like DeepSeek-R1 and OpenAI-o1 have demonstrated remarkable reasoning capabilities, raising important questions about their biases in LLM-as-a-judge settings. We present a comprehensive benchmark comparing judging biases between LLMs and LRMs across both subjective preference-alignment datasets and objective fact-based datasets. Through investigation of bandwagon, authority, position, and distraction biases, we uncover four key findings: (1) despite their advanced reasoning capabilities, LRMs remain susceptible to the above biases; (2) LRMs demonstrate better robustness than LLMs specifically on fact-related datasets; (3) LRMs exhibit notable position bias, preferring options in later positions; and (4) we identify a novel "superficial reflection bias" where phrases mimicking reasoning (e.g., "wait, let me think...") significantly influence model judgments. To address these biases, we design and evaluate three mitigation strategies: specialized system prompts that reduce judging biases by up to 19\% in preference alignment datasets and 14\% in fact-related datasets, in-context learning that provides up to 27\% improvement on preference tasks but shows inconsistent results on factual tasks, and a self-reflection mechanism that reduces biases by up to 10\% in preference datasets and 16\% in fact-related datasets, with self-reflection proving particularly effective for LRMs. Our work provides crucial insights for developing more reliable LLM-as-a-Judge frameworks, especially as LRMs become increasingly deployed as automated judges.

Related papers

• JudgeLRM: Large Reasoning Models as a Judge [65.14085339820795]
  We investigate whether Large Language Models (LLMs) judges truly benefit from enhanced reasoning capabilities.<n>We introduce JudgeLRM, a family of judgment-oriented LLMs trained using reinforcement learning (RL) with judge-wise, outcome-driven rewards.
  arXiv  Detail & Related papers  (2025-03-31T02:18:51Z)
• R-PRM: Reasoning-Driven Process Reward Modeling [53.06844294668382]
  Process Reward Models (PRMs) have emerged as a promising solution by evaluating each reasoning step.<n>Existing PRMs typically output evaluation scores directly, limiting both learning efficiency and evaluation accuracy.<n>We propose Reasoning-Driven Process Reward Modeling (R-PRM)<n>R-PRM generates seed data from limited annotations, effectively bootstrapping our model's reasoning capabilities.
  arXiv  Detail & Related papers  (2025-03-27T09:23:08Z)
• Preference Leakage: A Contamination Problem in LLM-as-a-judge [69.96778498636071]
  Large Language Models (LLMs) as judges and LLM-based data synthesis have emerged as two fundamental LLM-driven data annotation methods. In this work, we expose preference leakage, a contamination problem in LLM-as-a-judge caused by the relatedness between the synthetic data generators and LLM-based evaluators.
  arXiv  Detail & Related papers  (2025-02-03T17:13:03Z)
• RAZOR: Sharpening Knowledge by Cutting Bias with Unsupervised Text Rewriting [16.633948320306832]
  biases prevalent in manually constructed datasets can introduce spurious correlations between tokens and labels.<n>Existing debiasing methods often rely on prior knowledge of specific dataset biases.<n>We propose RAZOR, a novel, unsupervised, and data-focused debiasing approach based on text rewriting for shortcut mitigation.
  arXiv  Detail & Related papers  (2024-12-10T17:02:58Z)
• LLMs are Biased Evaluators But Not Biased for Retrieval Augmented Generation [28.61326111959728]
  Large language models (LLMs) exhibit significant biases in evaluation tasks, particularly in preferentially rating and favoring self-generated content. Our study addresses this knowledge gap by simulating two critical phases of the retrieval-augmented generation (RAG) framework. Contrary to previous findings, our results reveal no significant self-preference effect in RAG frameworks.
  arXiv  Detail & Related papers  (2024-10-28T08:32:09Z)
• Bias Similarity Across Large Language Models [32.0365189539138]
  We analyze bias through output distribution across multiple dimensions using two datasets (4K and 1M questions) Our results show that fine-tuning has minimal impact on output distributions, and proprietary models tend to overly response as unknowns to minimize bias, compromising accuracy and utility. Open-source models like Llama3-Chat and Gemma2-it demonstrate fairness comparable to proprietary models like GPT-4, challenging the assumption that larger, closed-source models are inherently less biased.
  arXiv  Detail & Related papers  (2024-10-15T19:21:14Z)
• Reward-Augmented Data Enhances Direct Preference Alignment of LLMs [63.32585910975191]
  We introduce reward-conditioned Large Language Models (LLMs) that learn from the entire spectrum of response quality within the dataset.<n>We propose an effective yet simple data relabeling method that conditions the preference pairs on quality scores to construct a reward-augmented dataset.
  arXiv  Detail & Related papers  (2024-10-10T16:01:51Z)
• Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge [84.34545223897578]
  Despite their excellence in many domains, potential issues are under-explored, undermining their reliability and the scope of their utility. We identify 12 key potential biases and propose a new automated bias quantification framework-CALM- which quantifies and analyzes each type of bias in LLM-as-a-Judge. Our work highlights the need for stakeholders to address these issues and remind users to exercise caution in LLM-as-a-Judge applications.
  arXiv  Detail & Related papers  (2024-10-03T17:53:30Z)
• Unboxing Occupational Bias: Grounded Debiasing of LLMs with U.S. Labor Data [9.90951705988724]
  Large Language Models (LLM) are prone to inheriting and amplifying societal biases. LLM bias can have far-reaching consequences, leading to unfair practices and exacerbating social inequalities.
  arXiv  Detail & Related papers  (2024-08-20T23:54:26Z)
• Interpreting Bias in Large Language Models: A Feature-Based Approach [0.0]
  Large Language Models (LLMs) have showcased remarkable performance across various natural language processing (NLP) tasks. This paper investigates the propagation of biases within LLMs through a novel feature-based analytical approach.
  arXiv  Detail & Related papers  (2024-06-18T07:28:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.