HumanRankEval: Automatic Evaluation of LMs as Conversational Assistants

• URL: http://arxiv.org/abs/2405.09186v1
• Date: Wed, 15 May 2024 08:47:26 GMT
• Title: HumanRankEval: Automatic Evaluation of LMs as Conversational Assistants
• Authors: Milan Gritta, Gerasimos Lampouras, Ignacio Iacobacci,
• Abstract summary: We propose a novel automatic evaluation task: HumanRankEval. It consists of a large-scale, diverse and high-quality set of questions, each with several answers authored and scored by humans. We show that HRE correlates well with human judgements and is particularly responsive to model changes following instruction-tuning.
• Score: 16.932009464531742
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models (LMs) as conversational assistants recently became popular tools that help people accomplish a variety of tasks. These typically result from adapting LMs pretrained on general domain text sequences through further instruction-tuning and possibly preference optimisation methods. The evaluation of such LMs would ideally be performed using human judgement, however, this is not scalable. On the other hand, automatic evaluation featuring auxiliary LMs as judges and/or knowledge-based tasks is scalable but struggles with assessing conversational ability and adherence to instructions. To help accelerate the development of LMs as conversational assistants, we propose a novel automatic evaluation task: HumanRankEval (HRE). It consists of a large-scale, diverse and high-quality set of questions, each with several answers authored and scored by humans. To perform evaluation, HRE ranks these answers based on their log-likelihood under the LM's distribution, and subsequently calculates their correlation with the corresponding human rankings. We support HRE's efficacy by investigating how efficiently it separates pretrained and instruction-tuned LMs of various sizes. We show that HRE correlates well with human judgements and is particularly responsive to model changes following instruction-tuning.

Related papers

• SedarEval: Automated Evaluation using Self-Adaptive Rubrics [4.97150240417381]
  We propose a new evaluation paradigm based on self-adaptive rubrics. SedarEval consists of 1,000 meticulously crafted questions, each with its own self-adaptive rubric. We train a specialized evaluator language model (evaluator LM) to supplant human graders.
  arXiv  Detail & Related papers  (2025-01-26T16:45:09Z)
• LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts [44.42036618863284]
  This paper introduces a framework for the automated evaluation of natural language texts. A large language model (LLM) is prompted with each rubric question and produces a distribution over potential responses. LLM predictions often fail to agree well with human judges -- indeed, the humans do not fully agree with one another. LLM-Rubric accomplishes this by training a small feed-forward neural network that includes both judge-specific and judge-independent parameters.
  arXiv  Detail & Related papers  (2024-12-31T04:57:01Z)
• HREF: Human Response-Guided Evaluation of Instruction Following in Language Models [61.273153125847166]
  We develop a new evaluation benchmark, Human Response-Guided Evaluation of Instruction Following (HREF) In addition to providing reliable evaluation, HREF emphasizes individual task performance and is free from contamination. We study the impact of key design choices in HREF, including the size of the evaluation set, the judge model, the baseline model, and the prompt template.
  arXiv  Detail & Related papers  (2024-12-20T03:26:47Z)
• A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look [52.114284476700874]
  This paper reports on the results of a large-scale evaluation (the TREC 2024 RAG Track) where four different relevance assessment approaches were deployed. We find that automatically generated UMBRELA judgments can replace fully manual judgments to accurately capture run-level effectiveness. Surprisingly, we find that LLM assistance does not appear to increase correlation with fully manual assessments, suggesting that costs associated with human-in-the-loop processes do not bring obvious tangible benefits.
  arXiv  Detail & Related papers  (2024-11-13T01:12:35Z)
• CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolution [74.41064280094064]
  textbfJudger-1 is the first open-source textbfall-in-one judge LLM. CompassJudger-1 is a general-purpose LLM that demonstrates remarkable versatility. textbfJudgerBench is a new benchmark that encompasses various subjective evaluation tasks.
  arXiv  Detail & Related papers  (2024-10-21T17:56:51Z)
• IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answering [10.338962367542331]
  We introduce an automatic evaluation framework IQA-EVAL to achieve Interactive Question Answering Evaluations. We also introduce a LLM-based Evaluation Agent (LEA) that can simulate human behaviors to generate interactions with IQA models. We show that our evaluation framework with GPT-4 as the backbone model achieves a high correlation with human evaluations on the IQA task.
  arXiv  Detail & Related papers  (2024-08-24T10:34:20Z)
• Psychometric Alignment: Capturing Human Knowledge Distributions via Language Models [41.324679754114165]
  Language models (LMs) are increasingly used to simulate human-like responses in scenarios where accurately mimicking a population's behavior can guide decision-making. We introduce "psychometric alignment," a metric that measures the extent to which LMs reflect human knowledge distribution. We find significant misalignment between LMs and human populations, though using persona-based prompts can improve alignment.
  arXiv  Detail & Related papers  (2024-07-22T14:02:59Z)
• Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussions [77.66677127535222]
  Auto-Arena is an innovative framework that automates the entire evaluation process using LLM-powered agents. In our experiments, Auto-Arena shows a 92.14% correlation with human preferences, surpassing all previous expert-annotated benchmarks.
  arXiv  Detail & Related papers  (2024-05-30T17:19:19Z)
• Beyond Static Evaluation: A Dynamic Approach to Assessing AI Assistants' API Invocation Capabilities [48.922660354417204]
  We propose Automated Dynamic Evaluation (AutoDE) to assess an assistant's API call capability without human involvement. In our framework, we endeavor to closely mirror genuine human conversation patterns in human-machine interactions.
  arXiv  Detail & Related papers  (2024-03-17T07:34:12Z)
• Towards LLM-based Autograding for Short Textual Answers [4.853810201626855]
  This manuscript is an evaluation of a large language model for the purpose of autograding. Our findings suggest that while "out-of-the-box" LLMs provide a valuable tool, their readiness for independent automated grading remains a work in progress.
  arXiv  Detail & Related papers  (2023-09-09T22:25:56Z)
• Can Large Language Models Be an Alternative to Human Evaluations? [80.81532239566992]
  Large language models (LLMs) have demonstrated exceptional performance on unseen tasks when only the task instructions are provided. We show that the result of LLM evaluation is consistent with the results obtained by expert human evaluation.
  arXiv  Detail & Related papers  (2023-05-03T07:28:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.