A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI

• URL: http://arxiv.org/abs/2412.13942v1
• Date: Wed, 18 Dec 2024 15:24:50 GMT
• Title: A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI
• Abstract summary: Disagreement in human labeling is ubiquitous, and can be captured in human judgment distributions (HJDs)<n>Recent research has shown that explanations provide valuable information for understanding human label variation (HLV)<n>This paper examines whether large language models (LLMs) can be used to replace humans in generating explanations for approximating HJD.
• Score: 41.029671072957264
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Disagreement in human labeling is ubiquitous, and can be captured in human judgment distributions (HJDs). Recent research has shown that explanations provide valuable information for understanding human label variation (HLV) and large language models (LLMs) can approximate HJD from a few human-provided label-explanation pairs. However, collecting explanations for every label is still time-consuming. This paper examines whether LLMs can be used to replace humans in generating explanations for approximating HJD. Specifically, we use LLMs as annotators to generate model explanations for a few given human labels. We test ways to obtain and combine these label-explanations with the goal to approximate human judgment distribution. We further compare the resulting human with model-generated explanations, and test automatic and human explanation selection. Our experiments show that LLM explanations are promising for NLI: to estimate HJD, generated explanations yield comparable results to human's when provided with human labels. Importantly, our results generalize from datasets with human explanations to i) datasets where they are not available and ii) challenging out-of-distribution test sets.

Related papers

• Text Annotation via Inductive Coding: Comparing Human Experts to LLMs in Qualitative Data Analysis [44.08932633077333]
  The study evaluates the performance of six open-source large language models (LLMs) compared to human experts.<n>The results reveal a peculiar dichotomy: human coders consistently perform well when labeling complex sentences but struggle with simpler ones, while LLMs exhibit the opposite trend.
  arXiv  Detail & Related papers  (2025-11-17T13:03:27Z)
• Threading the Needle: Reweaving Chain-of-Thought Reasoning to Explain Human Label Variation [44.25455164977285]
  Large Language Models (LLMs) generate chains of thought (CoTs) before giving the final answer.<n>We propose a novel pipeline enriched with linguistically-grounded discourse segmenters to extract supporting and opposing statements for each answer option.<n>We also propose a rank-based HLV evaluation framework that prioritizes the ranking of answers over exact scores.
  arXiv  Detail & Related papers  (2025-05-29T11:47:18Z)
• Comparing zero-shot self-explanations with human rationales in multilingual text classification [5.32539007352208]
  Instruction-tuned LLMs generate self-explanations that do not require computations or the application of possibly complex XAI methods. We analyse whether this ability results in a good explanation by evaluating self-explanations in the form of input rationales. Our results show that self-explanations align more closely with human annotations compared to LRP, while maintaining a comparable level of faithfulness.
  arXiv  Detail & Related papers  (2024-10-04T10:14:12Z)
• "Seeing the Big through the Small": Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations? [41.57673268042933]
  Large language models (LLMs) are increasingly used as evaluators but with mixed results. This study proposes to exploit LLMs to approximate HJDs using a small number of expert labels and explanations. Our experiments show that a few explanations significantly improve LLMs' ability to approximate HJDs with and without explicit labels.
  arXiv  Detail & Related papers  (2024-06-25T14:42:17Z)
• Evaluating Evidence Attribution in Generated Fact Checking Explanations [48.776087871960584]
  We introduce a novel evaluation protocol -- citation masking and recovery. We implement our protocol using both human annotators and automatic annotators. Experiments reveal that the best-performing LLMs still generate explanations with inaccurate attributions.
  arXiv  Detail & Related papers  (2024-06-18T14:13:13Z)
• XplainLLM: A QA Explanation Dataset for Understanding LLM Decision-Making [13.928951741632815]
  Large Language Models (LLMs) have recently made impressive strides in natural language understanding tasks. In this paper, we look into bringing some transparency to this process by introducing a new explanation dataset. Our dataset includes 12,102 question-answer-explanation (QAE) triples.
  arXiv  Detail & Related papers  (2023-11-15T00:34:28Z)
• Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations [62.61495090463084]
  Large language models (LLMs) are trained to imitate humans to explain human decisions. We evaluate whether an explanation can enable humans to precisely infer the model's outputs on diverse counterfactuals. We found that LLM's explanations have low precision and that precision does not correlate with plausibility.
  arXiv  Detail & Related papers  (2023-07-17T17:41:47Z)
• Using Natural Language Explanations to Rescale Human Judgments [81.66697572357477]
  We propose a method to rescale ordinal annotations and explanations using large language models (LLMs) We feed annotators' Likert ratings and corresponding explanations into an LLM and prompt it to produce a numeric score anchored in a scoring rubric. Our method rescales the raw judgments without impacting agreement and brings the scores closer to human judgments grounded in the same scoring rubric.
  arXiv  Detail & Related papers  (2023-05-24T06:19:14Z)
• Are Human Explanations Always Helpful? Towards Objective Evaluation of Human Natural Language Explanations [27.624182544486334]
  We build on the view that the quality of a human-annotated explanation can be measured based on its helpfulness. We define a new metric that can take into consideration the helpfulness of an explanation for model performance.
  arXiv  Detail & Related papers  (2023-05-04T19:31:50Z)
• Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language? [86.60613602337246]
  We introduce a leakage-adjusted simulatability (LAS) metric for evaluating NL explanations. LAS measures how well explanations help an observer predict a model's output, while controlling for how explanations can directly leak the output. We frame explanation generation as a multi-agent game and optimize explanations for simulatability while penalizing label leakage.
  arXiv  Detail & Related papers  (2020-10-08T16:59:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.