"Seeing the Big through the Small": Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations?

• URL: http://arxiv.org/abs/2406.17600v2
• Date: Fri, 04 Oct 2024 15:22:29 GMT
• Title: "Seeing the Big through the Small": Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations?
• Abstract summary: Large language models (LLMs) are increasingly used as evaluators but with mixed results. This study proposes to exploit LLMs to approximate HJDs using a small number of expert labels and explanations. Our experiments show that a few explanations significantly improve LLMs' ability to approximate HJDs with and without explicit labels.
• Score: 41.57673268042933
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human label variation (HLV) is a valuable source of information that arises when multiple human annotators provide different labels for valid reasons. In Natural Language Inference (NLI) earlier approaches to capturing HLV involve either collecting annotations from many crowd workers to represent human judgment distribution (HJD) or use expert linguists to provide detailed explanations for their chosen labels. While the former method provides denser HJD information, obtaining it is resource-intensive. In contrast, the latter offers richer textual information but it is challenging to scale up to many human judges. Besides, large language models (LLMs) are increasingly used as evaluators ("LLM judges") but with mixed results, and few works aim to study HJDs. This study proposes to exploit LLMs to approximate HJDs using a small number of expert labels and explanations. Our experiments show that a few explanations significantly improve LLMs' ability to approximate HJDs with and without explicit labels, thereby providing a solution to scale up annotations for HJD. However, fine-tuning smaller soft-label aware models with the LLM-generated model judgment distributions (MJDs) presents partially inconsistent results: while similar in distance, their resulting fine-tuned models and visualized distributions differ substantially. We show the importance of complementing instance-level distance measures with a global-level shape metric and visualization to more effectively evaluate MJDs against human judgment distributions.

Related papers

• Post-hoc Alignment of LLM-judges to Human Judgment Distribution [3.0607199893136006]
  We show that NAPHA consistently improves soft-labels prediction across base LLM models and datasets.<n>We also show via oracle experiments that improving entropy class prediction can substantially enhance NAPHA's practical effectiveness.
  arXiv  Detail & Related papers  (2026-09-01T11:03:56Z)
• Are Multimodal Large Language Models Good Annotators for Image Tagging? [62.01475514488922]
  This paper aims to analyze the gap between MLLM-generated and human annotations.<n>We propose TagLLM, a novel framework for image tagging, which aims to narrow the gap between MLLM-generated and human annotations.
  arXiv  Detail & Related papers  (2026-02-24T14:53:16Z)
• Blind to the Human Touch: Overlap Bias in LLM-Based Summary Evaluation [89.52571224447111]
  Large language model (LLM) judges have often been used alongside traditional, algorithm-based metrics for tasks like summarization.<n>We provide an LLM judge bias analysis as a function of overlap with human-written responses in the domain of summarization.
  arXiv  Detail & Related papers  (2026-02-07T19:39:28Z)
• Prompt Candidates, then Distill: A Teacher-Student Framework for LLM-driven Data Annotation [35.1208076670736]
  We propose a novel candidate annotation paradigm wherein large language models are encouraged to output all possible labels when incurring uncertainty.<n>To ensure unique labels are provided for downstream tasks, we develop a teacher-student framework CanDist that distills candidate annotations with a Small Language Model.
  arXiv  Detail & Related papers  (2025-06-04T11:42:37Z)
• A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI [41.029671072957264]
  Disagreement in human labeling is ubiquitous, and can be captured in human judgment distributions (HJDs) Recent research has shown that explanations provide valuable information for understanding human label variation (HLV) This paper examines whether large language models (LLMs) can be used to replace humans in generating explanations for approximating HJD.
  arXiv  Detail & Related papers  (2024-12-18T15:24:50Z)
• Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models [36.983534612895156]
  In the recent past, a popular way of evaluating natural language understanding (NLU) was to consider a model's ability to perform natural language inference (NLI) tasks. This paper focuses on five different NLI benchmarks across six models of different scales. We investigate if they are able to discriminate models of different size and quality and how their accuracies develop during training.
  arXiv  Detail & Related papers  (2024-11-21T13:09:36Z)
• Can We Use Large Language Models to Fill Relevance Judgment Holes? [9.208308067952155]
  We take initial steps towards extending existing test collections by employing Large Language Models (LLM) to fill the holes. We find substantially lower correlates when human plus automatic judgments are used.
  arXiv  Detail & Related papers  (2024-05-09T07:39:19Z)
• Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment [84.32768080422349]
  Alignment with human preference prevents large language models from generating misleading or toxic content. We propose a new formulation of prompt diversity, implying a linear correlation with the final performance of LLMs after fine-tuning.
  arXiv  Detail & Related papers  (2024-03-17T07:08:55Z)
• CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation [94.59630161324013]
  We propose CoAnnotating, a novel paradigm for Human-LLM co-annotation of unstructured texts at scale. Our empirical study shows CoAnnotating to be an effective means to allocate work from results on different datasets, with up to 21% performance improvement over random baseline.
  arXiv  Detail & Related papers  (2023-10-24T08:56:49Z)
• Using Natural Language Explanations to Rescale Human Judgments [81.66697572357477]
  We propose a method to rescale ordinal annotations and explanations using large language models (LLMs) We feed annotators' Likert ratings and corresponding explanations into an LLM and prompt it to produce a numeric score anchored in a scoring rubric. Our method rescales the raw judgments without impacting agreement and brings the scores closer to human judgments grounded in the same scoring rubric.
  arXiv  Detail & Related papers  (2023-05-24T06:19:14Z)
• Can Large Language Models Capture Dissenting Human Voices? [7.668954669688971]
  Large language models (LLMs) have shown impressive achievements in solving a broad range of tasks. We evaluate the performance and alignment of LLM distribution with humans using two different techniques. We show LLMs exhibit limited ability in solving NLI tasks and simultaneously fail to capture human disagreement distribution.
  arXiv  Detail & Related papers  (2023-05-23T07:55:34Z)
• Characterizing Attribution and Fluency Tradeoffs for Retrieval-Augmented Large Language Models [6.425088990363101]
  We examine the relationship between fluency and attribution in Large Language Models prompted with retrieved evidence. We show that larger models tend to do much better in both fluency and attribution. We propose a recipe that could allow smaller models to both close the gap with larger models and preserve the benefits of top-k retrieval.
  arXiv  Detail & Related papers  (2023-02-11T02:43:34Z)
• Benchmarking Large Language Models for News Summarization [79.37850439866938]
  Large language models (LLMs) have shown promise for automatic summarization but the reasons behind their successes are poorly understood. We find instruction tuning, and not model size, is the key to the LLM's zero-shot summarization capability.
  arXiv  Detail & Related papers  (2023-01-31T18:46:19Z)
• Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoning [76.17436599516074]
  We introduce distributed NLI, a new NLU task with a goal to predict the distribution of human judgements for natural language inference. We show that models can capture human judgement distribution by applying additional distribution estimation methods, namely, Monte Carlo (MC) Dropout, Deep Ensemble, Re-Calibration, and Distribution Distillation.
  arXiv  Detail & Related papers  (2021-04-18T01:25:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.