Exploring the Effects of Alignment on Numerical Bias in Large Language Models

• URL: http://arxiv.org/abs/2601.16444v2
• Date: Mon, 26 Jan 2026 02:24:36 GMT
• Title: Exploring the Effects of Alignment on Numerical Bias in Large Language Models
• Authors: Ayako Sato, Hwichan Kim, Zhousi Chen, Masato Mita, Mamoru Komachi,
• Abstract summary: "LLM-as-a-judge," which utilizes large language models (LLMs) as evaluators, has proven effective in many evaluation tasks.<n>This study investigates the cause of numerical bias in evaluators.
• Score: 14.918747967803734
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: "LLM-as-a-judge," which utilizes large language models (LLMs) as evaluators, has proven effective in many evaluation tasks. However, evaluator LLMs exhibit numerical bias, a phenomenon where certain evaluation scores are generated disproportionately often, leading reduced evaluation performance. This study investigates the cause of this bias. Given that most evaluator LLMs are aligned through instruction tuning and preference tuning, and that prior research suggests alignment reduces output diversity, we hypothesize that numerical bias arises from alignment. To test this, we compare outputs from pre- and post-alignment LLMs, and observe that alignment indeed increases numerical bias. We also explore mitigation strategies for post-alignment LLMs, including temperature scaling, distribution calibration, and score range adjustment. Among these, score range adjustment is most effective in reducing bias and improving performance, though still heuristic. Our findings highlight the need for further work on optimal score range selection and more robust mitigation strategies.

Related papers

• Evaluating Scoring Bias in LLM-as-a-Judge [8.67484421243584]
  Large Language Models (LLMs) are employed as evaluators for complex tasks.<n>There are various biases within LLM-as-a-Judge, which adversely affect the fairness and reliability of judgments.
  arXiv  Detail & Related papers  (2025-06-27T15:25:23Z)
• Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation [57.380464382910375]
  We show that the choice of feedback protocol for evaluation can significantly affect evaluation reliability and induce systematic biases.<n>We find that generator models can flip preferences by embedding distractor features.<n>We offer recommendations for choosing feedback protocols based on dataset characteristics and evaluation objectives.
  arXiv  Detail & Related papers  (2025-04-20T19:05:59Z)
• CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judges [21.580762639442913]
  We introduce CalibraEval, a novel label-free method for mitigating selection bias during inference. CalibraEval reformulates debiasing as an optimization task aimed at adjusting observed prediction distributions to align with unbiased prediction distributions. We show that CalibraEval effectively mitigates selection bias and improves performance compared to existing debiasing methods.
  arXiv  Detail & Related papers  (2024-10-20T13:47:39Z)
• Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge [84.34545223897578]
  Despite their excellence in many domains, potential issues are under-explored, undermining their reliability and the scope of their utility. We identify 12 key potential biases and propose a new automated bias quantification framework-CALM- which quantifies and analyzes each type of bias in LLM-as-a-Judge. Our work highlights the need for stakeholders to address these issues and remind users to exercise caution in LLM-as-a-Judge applications.
  arXiv  Detail & Related papers  (2024-10-03T17:53:30Z)
• Post-hoc Reward Calibration: A Case Study on Length Bias [38.47276516266]
  Reward models (RMs) can develop biases by exploiting spurious correlations in their training data.<n>These biases can lead to incorrect output rankings, sub-optimal model evaluations, and the amplification of undesirable behaviours.<n>This paper addresses the challenge of correcting such biases without additional data and training.
  arXiv  Detail & Related papers  (2024-09-25T22:30:42Z)
• Eliminating Position Bias of Language Models: A Mechanistic Approach [119.34143323054143]
  Position bias has proven to be a prevalent issue of modern language models (LMs)<n>Our mechanistic analysis attributes the position bias to two components employed in nearly all state-of-the-art LMs: causal attention and relative positional encodings.<n>By eliminating position bias, models achieve better performance and reliability in downstream tasks, including LM-as-a-judge, retrieval-augmented QA, molecule generation, and math reasoning.
  arXiv  Detail & Related papers  (2024-07-01T09:06:57Z)
• Preference Learning Algorithms Do Not Learn Preference Rankings [62.335733662381884]
  We study the conventional wisdom that preference learning trains models to assign higher likelihoods to more preferred outputs than less preferred outputs. We find that most state-of-the-art preference-tuned models achieve a ranking accuracy of less than 60% on common preference datasets.
  arXiv  Detail & Related papers  (2024-05-29T21:29:44Z)
• Beyond Performance: Quantifying and Mitigating Label Bias in LLMs [8.77694178599322]
  We evaluate different approaches to quantifying label bias in a model's predictions. Our investigation reveals substantial label bias in models both before and after debiasing attempts. We propose a novel label bias calibration method tailored for few-shot prompting.
  arXiv  Detail & Related papers  (2024-05-04T19:53:03Z)
• The Gaps between Pre-train and Downstream Settings in Bias Evaluation and Debiasing [74.7319697510621]
  In-Context Learning (ICL) induces smaller changes to PLMs compared to FT-based debiasing methods. ICL-based debiasing methods show a higher correlation between intrinsic and extrinsic bias scores compared to FT-based methods.
  arXiv  Detail & Related papers  (2024-01-16T17:15:08Z)
• Self-Supervised Position Debiasing for Large Language Models [39.261233221850155]
  We propose a self-supervised position debiasing (SOD) framework to mitigate position bias for large language models (LLMs) Experiments on eight datasets and five tasks show that SOD consistently outperforms existing methods in mitigating three types of position biases.
  arXiv  Detail & Related papers  (2024-01-02T14:12:41Z)
• GPTBIAS: A Comprehensive Framework for Evaluating Bias in Large Language Models [83.30078426829627]
  Large language models (LLMs) have gained popularity and are being widely adopted by a large user community. The existing evaluation methods have many constraints, and their results exhibit a limited degree of interpretability. We propose a bias evaluation framework named GPTBIAS that leverages the high performance of LLMs to assess bias in models.
  arXiv  Detail & Related papers  (2023-12-11T12:02:14Z)
• The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning [61.68787689234622]
  A recent study, LIMA, shows that using merely 1K examples for alignment tuning can achieve significant alignment performance as well. This raises questions about how exactly the alignment tuning transforms a base LLM. We show that the gap between tuning-free and tuning-based alignment methods can be significantly reduced through strategic prompting.
  arXiv  Detail & Related papers  (2023-12-04T00:46:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.