From Fallback to Frontline: When Can LLMs be Superior Annotators of Human Perspectives?

• URL: http://arxiv.org/abs/2604.17968v1
• Date: Mon, 20 Apr 2026 08:48:38 GMT
• Title: From Fallback to Frontline: When Can LLMs be Superior Annotators of Human Perspectives?
• Abstract summary: Large language models (LLMs) are increasingly used as annotators at scale.<n>This work challenges that presumption by framing perspective-taking as the estimation of a latent group-level judgment.
• Score: 16.841321593524132
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Although large language models (LLMs) are increasingly used as annotators at scale, they are typically treated as a pragmatic fallback rather than a faithful estimator of human perspectives. This work challenges that presumption. By framing perspective-taking as the estimation of a latent group-level judgment, we characterize the conditions under which modern LLMs can outperform human annotators, including in-group humans, when predicting aggregate subgroup opinions on subjective tasks, and show that these conditions are common in practice. This advantage arises from structural properties of LLMs as estimators, including low variance and reduced coupling between representation and processing biases, rather than any claim of lived experience. Our analysis identifies clear regimes where LLMs act as statistically superior frontline estimators, as well as principled limits where human judgment remains essential. These findings reposition LLMs from a cost-saving compromise to a principled tool for estimating collective human perspectives.

Related papers

• Presupposition and Reasoning in Conditionals: A Theory-Based Study of Humans and LLMs [1.0599607477285324]
  We compare human judgments and LLM predictions on a normed dataset of conditional sentences.<n>Results show that humans integrate probabilistic and pragmatic cues in their judgment.<n>We observe models that best match human ratings often lack coherent pragmatic reasoning.
  arXiv  Detail & Related papers  (2026-05-18T13:08:20Z)
• Augmenting Human Evaluation with LLM Judges: How Many Human Reviews Do You Need? [1.111977509278551]
  Large language models (LLMs) are increasingly used as automated evaluators of AI systems, including in high-stakes applications.<n> Expert human ratings are costly and difficult to scale, whereas LLM ratings can be produced quickly at low cost.<n>This paper shifts the role of the LLM judge from substitutive to auxiliary, and formulates the LLM-as-a-judge paradigm as one of augmenting human evaluation.
  arXiv  Detail & Related papers  (2026-05-08T17:13:08Z)
• Evaluating Alignment of Behavioral Dispositions in LLMs [15.282965130762648]
  We focus on behavioral dispositions that shape responses in social contexts.<n>We introduce a framework to study how closely the dispositions expressed by LLMs align with those of humans.
  arXiv  Detail & Related papers  (2026-02-11T19:59:12Z)
• Automated Concept Discovery for LLM-as-a-Judge Preference Analysis [21.171990974350773]
  Large Language Models (LLMs) are increasingly used as scalable evaluators of model outputs.<n>Their preference judgments exhibit systematic biases and can diverge from human evaluations.<n>We study several embedding-level concept extraction methods for analyzing LLM judge behavior.
  arXiv  Detail & Related papers  (2026-02-09T20:55:16Z)
• On Evaluating LLM Alignment by Evaluating LLMs as Judges [68.15541137648721]
  evaluating large language models' (LLMs) alignment requires them to be helpful, honest, safe, and to precisely follow human instructions.<n>We examine the relationship between LLMs' generation and evaluation capabilities in aligning with human preferences.<n>We propose a benchmark that assesses alignment without directly evaluating model outputs.
  arXiv  Detail & Related papers  (2025-11-25T18:33:24Z)
• Arbiters of Ambivalence: Challenges of Using LLMs in No-Consensus Tasks [52.098988739649705]
  This study examines the biases and limitations of LLMs in three roles: answer generator, judge, and debater.<n>We develop a no-consensus'' benchmark by curating examples that encompass a variety of a priori ambivalent scenarios.<n>Our results show that while LLMs can provide nuanced assessments when generating open-ended answers, they tend to take a stance on no-consensus topics when employed as judges or debaters.
  arXiv  Detail & Related papers  (2025-05-28T01:31:54Z)
• Can DPO Learn Diverse Human Values? A Theoretical Scaling Law [7.374590753074647]
  Preference learning trains models to distinguish between preferred and non-preferred responses based on human feedback.<n>This paper introduces a new theoretical framework to analyze how generalization scales with value diversity and sample quantity.<n>Our framework rigorously assesses how well models generalize after a finite number of gradient steps.
  arXiv  Detail & Related papers  (2024-08-06T22:11:00Z)
• LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks [106.09361690937618]
  There is an increasing trend towards evaluating NLP models with LLMs instead of human judgments.<n>We provide JUDGE-BENCH, a collection of 20 NLP datasets with human annotations covering a broad range of evaluated properties and types of data.<n>We evaluate 11 current LLMs, covering both open-weight and proprietary models, for their ability to replicate the annotations.
  arXiv  Detail & Related papers  (2024-06-26T14:56:13Z)
• A Survey on Human Preference Learning for Large Language Models [81.41868485811625]
  The recent surge of versatile large language models (LLMs) largely depends on aligning increasingly capable foundation models with human intentions by preference learning. This survey covers the sources and formats of preference feedback, the modeling and usage of preference signals, as well as the evaluation of the aligned LLMs.
  arXiv  Detail & Related papers  (2024-06-17T03:52:51Z)
• Bayesian Statistical Modeling with Predictors from LLMs [5.5711773076846365]
  State of the art large language models (LLMs) have shown impressive performance on a variety of benchmark tasks. This raises questions about the human-likeness of LLM-derived information.
  arXiv  Detail & Related papers  (2024-06-13T11:33:30Z)
• Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks [65.69651759036535]
  We analyze whether large language models (LLMs) can serve as reliable alternatives to humans.<n>This paper explores both conventional tasks (e.g., story generation) and alignment tasks (e.g., math reasoning)<n>We find that LLM evaluators can generate unnecessary criteria or omit crucial criteria, resulting in a slight deviation from the experts.
  arXiv  Detail & Related papers  (2023-10-30T17:04:35Z)
• On Learning to Summarize with Large Language Models as References [101.79795027550959]
  Large language models (LLMs) are favored by human annotators over the original reference summaries in commonly used summarization datasets. We study an LLM-as-reference learning setting for smaller text summarization models to investigate whether their performance can be substantially improved.
  arXiv  Detail & Related papers  (2023-05-23T16:56:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.