Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge

• URL: http://arxiv.org/abs/2604.05593v1
• Date: Tue, 07 Apr 2026 08:43:30 GMT
• Title: Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge
• Abstract summary: Large language models (LLMs) are increasingly used as automated evaluators (LLM-as-a-Judge)<n>This work challenges its reliability by showing that trust judgments by LLMs are biased by disclosed source labels.<n>Eye-tracking data reveal that humans rely heavily on source labels as cues for judgments.
• Score: 26.31243351014906
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large language models (LLMs) are increasingly used as automated evaluators (LLM-as-a-Judge). This work challenges its reliability by showing that trust judgments by LLMs are biased by disclosed source labels. Using a counterfactual design, we find that both humans and LLM judges assign higher trust to information labeled as human-authored than to the same content labeled as AI-generated. Eye-tracking data reveal that humans rely heavily on source labels as heuristic cues for judgments. We analyze LLM internal states during judgment. Across label conditions, models allocate denser attention to the label region than the content region, and this label dominance is stronger under Human labels than AI labels, consistent with the human gaze patterns. Besides, decision uncertainty measured by logits is higher under AI labels than Human labels. These results indicate that the source label is a salient heuristic cue for both humans and LLMs. It raises validity concerns for label-sensitive LLM-as-a-Judge evaluation, and we cautiously raise that aligning models with human preferences may propagate human heuristic reliance into models, motivating debiased evaluation and alignment.

Related papers

• Label Over Logic? How Source Cues Bias Human Fallacy Judgments More Than LLMs [17.420529906099503]
  We evaluate whether source-label bias in reasoning evaluation is primarily a human vulnerability.<n>Our findings highlight the potential of human-LLM collaboration in increasingly AI-mediated environments.
  arXiv  Detail & Related papers  (2026-05-28T13:41:43Z)
• Text Annotation via Inductive Coding: Comparing Human Experts to LLMs in Qualitative Data Analysis [44.08932633077333]
  The study evaluates the performance of six open-source large language models (LLMs) compared to human experts.<n>The results reveal a peculiar dichotomy: human coders consistently perform well when labeling complex sentences but struggle with simpler ones, while LLMs exhibit the opposite trend.
  arXiv  Detail & Related papers  (2025-11-17T13:03:27Z)
• Towards Better IncomLDL: We Are Unaware of Hidden Labels in Advance [68.65348495515258]
  We presentHidLDL, which aims to recover a complete label distribution from a real-world incomplete label distribution.<n>We simultaneously use local feature similarity and the global low-rank structure to reveal the mysterious veil of hidden labels.
  arXiv  Detail & Related papers  (2025-11-16T08:10:26Z)
• Human-Corrected Labels Learning: Enhancing Labels Quality via Human Correction of VLMs Discrepancies [6.58446551781724]
  We propose Human-Corrected Labels (HCLs), a novel setting that efficient human correction for VLM-generated noisy labels.<n>HCLs deploys human correction only for instances with VLM discrepancies, achieving both higher-quality annotations and reduced labor costs.<n>Our approach achieves superior classification performance and is robust to label noise, validating the effectiveness of HCL in practical weak supervision scenarios.
  arXiv  Detail & Related papers  (2025-11-12T07:38:19Z)
• Label Distribution Learning with Biased Annotations by Learning Multi-Label Representation [120.97262070068224]
  Multi-label learning (MLL) has gained attention for its ability to represent real-world data.<n>Label Distribution Learning (LDL) faces challenges in collecting accurate label distributions.
  arXiv  Detail & Related papers  (2025-02-03T09:04:03Z)
• "All that Glitters": Approaches to Evaluations with Unreliable Model and Human Annotations [0.0]
  "Gold" and "ground truth" human-mediated labels have error. This study demonstrates methods for answering such questions even in the context of very low reliabilities from expert humans.
  arXiv  Detail & Related papers  (2024-11-23T19:18:08Z)
• Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge [51.93909886542317]
  We show how *relying on a single aggregate correlation score* can obscure fundamental differences between human labels and those from automatic evaluation.<n>We propose stratifying data by human label uncertainty to provide a more robust analysis of automatic evaluation performance.
  arXiv  Detail & Related papers  (2024-10-03T03:08:29Z)
• Bring Your Own Data! Self-Supervised Evaluation for Large Language Models [52.15056231665816]
  We propose a framework for self-supervised evaluation of Large Language Models (LLMs) We demonstrate self-supervised evaluation strategies for measuring closed-book knowledge, toxicity, and long-range context dependence. We find strong correlations between self-supervised and human-supervised evaluations.
  arXiv  Detail & Related papers  (2023-06-23T17:59:09Z)
• Label Noise-Resistant Mean Teaching for Weakly Supervised Fake News Detection [93.6222609806278]
  We propose a novel label noise-resistant mean teaching approach (LNMT) for weakly supervised fake news detection. LNMT leverages unlabeled news and feedback comments of users to enlarge the amount of training data. LNMT establishes a mean teacher framework equipped with label propagation and label reliability estimation.
  arXiv  Detail & Related papers  (2022-06-10T16:01:58Z)
• A Study on the Autoregressive and non-Autoregressive Multi-label Learning [77.11075863067131]
  We propose a self-attention based variational encoder-model to extract the label-label and label-feature dependencies jointly. Our model can therefore be used to predict all labels in parallel while still including both label-label and label-feature dependencies.
  arXiv  Detail & Related papers  (2020-12-03T05:41:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.