Humans or LLMs as the Judge? A Study on Judgement Biases

• URL: http://arxiv.org/abs/2402.10669v5
• Date: Thu, 26 Sep 2024 03:16:52 GMT
• Title: Humans or LLMs as the Judge? A Study on Judgement Biases
• Authors: Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, Benyou Wang,
• Abstract summary: We propose a novel framework that is free from referencing groundtruth annotations for investigating Misinformation Oversight Bias, Gender Bias, Authority Bias and Beauty Bias on LLM and human judges. Results show that human and LLM judges are vulnerable to perturbations to various degrees, and that even the cutting-edge judges possess considerable biases.
• Score: 17.069314000437537
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adopting human and large language models (LLM) as judges (a.k.a human- and LLM-as-a-judge) for evaluating the performance of LLMs has recently gained attention. Nonetheless, this approach concurrently introduces potential biases from human and LLMs, questioning the reliability of the evaluation results. In this paper, we propose a novel framework that is free from referencing groundtruth annotations for investigating Misinformation Oversight Bias, Gender Bias, Authority Bias and Beauty Bias on LLM and human judges. We curate a dataset referring to the revised Bloom's Taxonomy and conduct thousands of evaluations. Results show that human and LLM judges are vulnerable to perturbations to various degrees, and that even the cutting-edge judges possess considerable biases. We further exploit these biases to conduct attacks on LLM judges. We hope that our work can notify the community of the bias and vulnerability of human- and LLM-as-a-judge, as well as the urgency of developing robust evaluation systems.

Related papers

• Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review [66.73247554182376]
  Large language models (LLMs) have led to their integration into peer review. The unchecked adoption of LLMs poses significant risks to the integrity of the peer review system. We show that manipulating 5% of the reviews could potentially cause 12% of the papers to lose their position in the top 30% rankings.
  arXiv  Detail & Related papers  (2024-12-02T16:55:03Z)
• From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge [32.55871325700294]
  Assessment and evaluation have long been critical challenges in artificial intelligence (AI) and natural language processing (NLP) Recent advancements in Large Language Models (LLMs) inspire the "LLM-as-a-judge" paradigm.
  arXiv  Detail & Related papers  (2024-11-25T17:28:44Z)
• JudgeBench: A Benchmark for Evaluating LLM-based Judges [61.048125269475854]
  JudgeBench is a benchmark for evaluating LLM-based judges on challenging response pairs spanning knowledge, reasoning, math, and coding. Our comprehensive evaluation on a collection of prompted judges, fine-tuned judges, multi-agent judges, and reward models shows that JudgeBench poses a significantly greater challenge than previous benchmarks.
  arXiv  Detail & Related papers  (2024-10-16T17:58:19Z)
• Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge [84.34545223897578]
  Despite their excellence in many domains, potential issues are under-explored, undermining their reliability and the scope of their utility. We identify 12 key potential biases and propose a new automated bias quantification framework-CALM- which quantifies and analyzes each type of bias in LLM-as-a-Judge. Our work highlights the need for stakeholders to address these issues and remind users to exercise caution in LLM-as-a-Judge applications.
  arXiv  Detail & Related papers  (2024-10-03T17:53:30Z)
• Mitigating the Bias of Large Language Model Evaluation [30.67730115141905]
  We propose systematic research about the bias of LLM-as-a-Judge. For closed-source judge models, we apply calibration to mitigate the significance of superficial quality. For open-source judge models, we propose to mitigate the bias by contrastive training, with curated negative samples that deviate from instruction but present better superficial quality.
  arXiv  Detail & Related papers  (2024-09-25T09:52:44Z)
• Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates [10.091146498861333]
  Commercial large language models (LLMs) like GPT-4 have been recently employed to evaluate and compare different alignment approaches. We develop a framework to evaluate, compare, and visualize the reliability and alignment of LLM judges.
  arXiv  Detail & Related papers  (2024-08-23T11:49:01Z)
• Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructions [18.93335792080899]
  We investigate how much influence prompting the LLMs-as-a-judge has on the alignment of AI judgements to human judgements. We aggregate a taxonomy of quality criteria commonly used across state-of-the-art evaluations with LLMs and provide this as a rigorous benchmark of models as judges.
  arXiv  Detail & Related papers  (2024-08-16T14:49:35Z)
• LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks [106.09361690937618]
  There is an increasing trend towards evaluating NLP models with LLMs instead of human judgments. We provide JUDGE-BENCH, a collection of 20 NLP datasets with human annotations covering a broad range of evaluated properties and types of data. We evaluate 11 current LLMs, covering both open-weight and proprietary models, for their ability to replicate the annotations.
  arXiv  Detail & Related papers  (2024-06-26T14:56:13Z)
• JudgeLM: Fine-tuned Large Language Models are Scalable Judges [54.007823006976516]
  We propose to fine-tune Large Language Models (LLMs) as scalable judges (JudgeLM) to evaluate LLMs efficiently and effectively in open-ended benchmarks. We first propose a comprehensive, large-scale, high-quality dataset containing task seeds, LLMs-generated answers, and GPT-4-generated judgments for fine-tuning high-performance judges. We then analyze the key biases in fine-tuning LLM as a judge and consider them as position bias, knowledge bias, and format bias.
  arXiv  Detail & Related papers  (2023-10-26T17:48:58Z)
• Can Large Language Models Be an Alternative to Human Evaluations? [80.81532239566992]
  Large language models (LLMs) have demonstrated exceptional performance on unseen tasks when only the task instructions are provided. We show that the result of LLM evaluation is consistent with the results obtained by expert human evaluation.
  arXiv  Detail & Related papers  (2023-05-03T07:28:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.