Wisdom of Instruction-Tuned Language Model Crowds. Exploring Model Label Variation

• URL: http://arxiv.org/abs/2307.12973v2
• Date: Mon, 15 Apr 2024 09:00:26 GMT
• Title: Wisdom of Instruction-Tuned Language Model Crowds. Exploring Model Label Variation
• Abstract summary: Large Language Models (LLMs) exhibit remarkable text classification capabilities. Recent studies emphasize the importance of considering human label variation in data annotation. We evaluate four recent instruction-tuned LLMs as annotators on five subjective tasks across four languages.
• Score: 37.14258903995738
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) exhibit remarkable text classification capabilities, excelling in zero- and few-shot learning (ZSL and FSL) scenarios. However, since they are trained on different datasets, performance varies widely across tasks between those models. Recent studies emphasize the importance of considering human label variation in data annotation. However, how this human label variation also applies to LLMs remains unexplored. Given this likely model specialization, we ask: Do aggregate LLM labels improve over individual models (as for human annotators)? We evaluate four recent instruction-tuned LLMs as annotators on five subjective tasks across four languages. We use ZSL and FSL setups and label aggregation from human annotation. Aggregations are indeed substantially better than any individual model, benefiting from specialization in diverse tasks or languages. Surprisingly, FSL does not surpass ZSL, as it depends on the quality of the selected examples. However, there seems to be no good information-theoretical strategy to select those. We find that no LLM method rivals even simple supervised models. We also discuss the tradeoffs in accuracy, cost, and moral/ethical considerations between LLM and human annotation.

Related papers

• Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspective [27.35951269350466]
  Semi-supervised few-shot learning formulates real-world applications like ''auto-annotation''<n>We propose Stage-Wise Finetuning with Temperature Tuning (SWIFT), which enables existing SSL methods to effectively finetune a VLM on limited labeled data, abundant unlabeled data, and task-relevant but noisy data retrieved from the VLM's pretraining set.
  arXiv  Detail & Related papers  (2025-12-11T03:06:16Z)
• Can LLM Annotations Replace User Clicks for Learning to Rank? [112.2254432364736]
  Large-scale supervised data is essential for training modern ranking models, but obtaining high-quality human annotations is costly.<n>Click data has been widely used as a low-cost alternative, and with recent advances in large language models (LLMs), LLM-based relevance annotation has emerged as another promising annotation.<n> Experiments on both a public dataset, TianGong-ST, and an industrial dataset, Baidu-Click, show that click-supervised models perform better on high-frequency queries.<n>We explore two training strategies -- data scheduling and frequency-aware multi-objective learning -- that integrate both supervision signals.
  arXiv  Detail & Related papers  (2025-11-10T02:26:14Z)
• Bridging the Gap: In-Context Learning for Modeling Human Disagreement [8.011316959982654]
  Large Language Models (LLMs) have shown strong performance on NLP classification tasks.<n>This study examines whether LLMs can capture multiple perspectives and reflect annotator disagreement in subjective tasks such as hate speech and offensive language detection.
  arXiv  Detail & Related papers  (2025-06-06T14:24:29Z)
• Evaluating how LLM annotations represent diverse views on contentious topics [3.405231040967506]
  We show how generative large language models (LLMs) represent diverse viewpoints on contentious labeling tasks. Our findings suggest that when using LLMs to annotate data, under-representing the views of particular groups is not a substantial concern.
  arXiv  Detail & Related papers  (2025-03-29T22:53:15Z)
• Idiosyncrasies in Large Language Models [54.26923012617675]
  We unveil and study idiosyncrasies in Large Language Models (LLMs) We find that fine-tuning existing text embedding models on LLM-generated texts yields excellent classification accuracy. We leverage LLM as judges to generate detailed, open-ended descriptions of each model's idiosyncrasies.
  arXiv  Detail & Related papers  (2025-02-17T18:59:02Z)
• Formality is Favored: Unraveling the Learning Preferences of Large Language Models on Data with Conflicting Knowledge [55.65162959527848]
  Large language models have shown excellent performance on many knowledge-intensive tasks. However, pretraining data tends to contain misleading and even conflicting information. This study systematically analyze LLMs' learning preferences for data with conflicting knowledge.
  arXiv  Detail & Related papers  (2024-10-07T06:49:41Z)
• On Unsupervised Prompt Learning for Classification with Black-box Language Models [71.60563181678323]
  Large language models (LLMs) have achieved impressive success in text-formatted learning problems. LLMs can label datasets with even better quality than skilled human annotators. In this paper, we propose unsupervised prompt learning for classification with black-box LLMs.
  arXiv  Detail & Related papers  (2024-10-04T03:39:28Z)
• Zero-to-Strong Generalization: Eliciting Strong Capabilities of Large Language Models Iteratively without Gold Labels [75.77877889764073]
  Large Language Models (LLMs) have demonstrated remarkable performance through supervised fine-tuning or in-context learning using gold labels. This study explores whether solely utilizing unlabeled data can elicit strong model capabilities. We propose a new paradigm termed zero-to-strong generalization.
  arXiv  Detail & Related papers  (2024-09-19T02:59:44Z)
• Leveraging Weakly Annotated Data for Hate Speech Detection in Code-Mixed Hinglish: A Feasibility-Driven Transfer Learning Approach with Large Language Models [0.0]
  Hate speech detection in mix-code low-resource languages is an active problem area where the use of Large Language Models has proven beneficial. In this study, we have compiled a dataset of 100 YouTube comments, and weakly labelled them for coarse and fine-grained misogyny classification in mix-code Hinglish. Out of all the approaches, zero-shot classification using the Bidirectional Auto-Regressive Transformers (BART) large model and few-shot prompting using Generative Pre-trained Transformer- 3 (ChatGPT-3) achieve the best results.
  arXiv  Detail & Related papers  (2024-03-04T15:27:49Z)
• Customizing Language Model Responses with Contrastive In-Context Learning [7.342346948935483]
  We propose an approach that uses contrastive examples to better describe our intent. This involves providing positive examples that illustrate the true intent, along with negative examples that show what characteristics we want LLMs to avoid. Before generating an answer, we ask the model to analyze the examples to teach itself what to avoid. This reasoning step provides the model with the appropriate articulation of the user's need and guides it towards generting a better answer.
  arXiv  Detail & Related papers  (2024-01-30T19:13:12Z)
• How Robust are LLMs to In-Context Majority Label Bias? [3.3577727874594654]
  We study the robustness of in-context learning in Large Language Models (LLMs) to shifts that occur due to majority label bias. Our findings also highlight the impact of model size and the richness of instructional prompts contributing towards model robustness.
  arXiv  Detail & Related papers  (2023-12-27T12:20:12Z)
• CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation [94.59630161324013]
  We propose CoAnnotating, a novel paradigm for Human-LLM co-annotation of unstructured texts at scale. Our empirical study shows CoAnnotating to be an effective means to allocate work from results on different datasets, with up to 21% performance improvement over random baseline.
  arXiv  Detail & Related papers  (2023-10-24T08:56:49Z)
• Verbosity Bias in Preference Labeling by Large Language Models [10.242500241407466]
  We examine the biases that come along with evaluating Large Language Models (LLMs) We take a closer look into verbosity bias -- a bias where LLMs sometimes prefer more verbose answers even if they have similar qualities.
  arXiv  Detail & Related papers  (2023-10-16T05:19:02Z)
• Large Language Models Vote: Prompting for Rare Disease Identification [3.2063918860622795]
  We propose Models-Vote Prompting (MVP), a flexible prompting approach for improving the performance of generative Large Language Models (LLMs) MVP works by prompting numerous LLMs to perform the same tasks and conducting a majority vote on the resulting outputs. We also release a novel rare disease dataset for FSL, available to those who signed the MIMIC-IV Data Use Agreement (DUA)
  arXiv  Detail & Related papers  (2023-08-24T16:09:13Z)
• Revisiting Self-Training for Few-Shot Learning of Language Model [61.173976954360334]
  Unlabeled data carry rich task-relevant information, they are proven useful for few-shot learning of language model. In this work, we revisit the self-training technique for language model fine-tuning and present a state-of-the-art prompt-based few-shot learner, SFLM.
  arXiv  Detail & Related papers  (2021-10-04T08:51:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.