Label Over Logic? How Source Cues Bias Human Fallacy Judgments More Than LLMs

• URL: http://arxiv.org/abs/2605.29928v2
• Date: Wed, 03 Jun 2026 17:39:58 GMT
• Title: Label Over Logic? How Source Cues Bias Human Fallacy Judgments More Than LLMs
• Abstract summary: We evaluate whether source-label bias in reasoning evaluation is primarily a human vulnerability.<n>Our findings highlight the potential of human-LLM collaboration in increasingly AI-mediated environments.
• Score: 17.420529906099503
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As AI-generated and AI-assisted content floods online spaces, source labels attached to such content can distort human reasoning judgments, with downstream consequences for moderation, evaluation, and decision-making. Whether LLMs share this vulnerability, or offer more source-agnostic evaluation, remains an open question with direct implications for human-AI collaboration. We examine this issue using logical fallacies as a controlled setting to isolate source-label effects on reasoning quality, independent of domain knowledge. We conduct an online study (N=505) where participants are assigned to a source condition (human, AI, human with AI assistance, AI with human assistance, or no disclosure) and evaluate comments containing logical fallacies, comparing their judgments with those of LLMs (GPT-5.2, Gemini 2.5 Flash, Claude Sonnet 4.5), who were evaluated across the same source conditions. Human evaluators were significantly more susceptible to fallacies labeled as written by human or human with AI assistance and assigned higher trust and evaluation ratings in these conditions. LLM evaluations remained comparatively stable across source labels, though performance varied across models. Confidence levels were similarly high across conditions for both humans and LLMs, regardless of fallacy presence. Our findings indicate that source-label bias in reasoning evaluation is primarily a human vulnerability and highlight the potential of human-LLM collaboration in increasingly AI-mediated environments.

Related papers

• Thesis Proposal: Toward a Human-Centered and Perspective-Aware Framework for Reproducible ML Evaluation and AI Alignment [4.886462623872524]
  It is essential to take disagreement into account when building human-centered AI systems.<n>Disagreement often arises from subjective human opinion and can vary with one's identity, beliefs, and social environment.<n>We present a human-centered and perspective-aware framework for reproducible ML evaluation and AI alignment.
  arXiv  Detail & Related papers  (2026-08-31T14:12:23Z)
• Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge [26.31243351014906]
  Large language models (LLMs) are increasingly used as automated evaluators (LLM-as-a-Judge)<n>This work challenges its reliability by showing that trust judgments by LLMs are biased by disclosed source labels.<n>Eye-tracking data reveal that humans rely heavily on source labels as cues for judgments.
  arXiv  Detail & Related papers  (2026-04-07T08:43:30Z)
• When Should Humans Step In? Optimal Human Dispatching in AI-Assisted Decisions [2.7755032786791003]
  We propose a general decision-theoretic framework for human-AI collaboration.<n>We treat AI assessments as factor-level signals and human judgments as costly information that can be selectively acquired.<n>We apply our framework to AI-assisted peer review.
  arXiv  Detail & Related papers  (2026-03-14T01:47:38Z)
• The AI Imperative: Scaling High-Quality Peer Review in Machine Learning [49.87236114682497]
  We argue that AI-assisted peer review must become an urgent research and infrastructure priority.<n>We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making.
  arXiv  Detail & Related papers  (2025-06-09T18:37:14Z)
• Measurement of LLM's Philosophies of Human Nature [113.47929131143766]
  We design the standardized psychological scale specifically targeting large language models (LLM)<n>We show that current LLMs exhibit a systemic lack of trust in humans.<n>We propose a mental loop learning framework, which enables LLM to continuously optimize its value system.
  arXiv  Detail & Related papers  (2025-04-03T06:22:19Z)
• Human aversion? Do AI Agents Judge Identity More Harshly Than Performance [0.06554326244334868]
  We investigate how AI agents based on large language models assess and integrate human input.<n>We find that the AI system systematically discounts human advice, penalizing human errors more severely than algorithmic errors.
  arXiv  Detail & Related papers  (2025-03-31T02:05:27Z)
• On Benchmarking Human-Like Intelligence in Machines [77.55118048492021]
  We argue that current AI evaluation paradigms are insufficient for assessing human-like cognitive capabilities.<n>We identify a set of key shortcomings: a lack of human-validated labels, inadequate representation of human response variability and uncertainty, and reliance on simplified and ecologically-invalid tasks.
  arXiv  Detail & Related papers  (2025-02-27T20:21:36Z)
• The Decoy Dilemma in Online Medical Information Evaluation: A Comparative Study of Credibility Assessments by LLM and Human Judges [4.65004369765875]
  It is not clear to what extent large language models (LLMs) behave "rationally" Our study empirically confirms the cognitive bias risks embedded in LLM agents. It highlights the complexity and importance of debiasing AI agents.
  arXiv  Detail & Related papers  (2024-11-23T00:43:27Z)
• Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge [51.93909886542317]
  We show how *relying on a single aggregate correlation score* can obscure fundamental differences between human labels and those from automatic evaluation.<n>We propose stratifying data by human label uncertainty to provide a more robust analysis of automatic evaluation performance.
  arXiv  Detail & Related papers  (2024-10-03T03:08:29Z)
• Human Bias in the Face of AI: Examining Human Judgment Against Text Labeled as AI Generated [48.70176791365903]
  This study explores how bias shapes the perception of AI versus human generated content.<n>We investigated how human raters respond to labeled and unlabeled content.
  arXiv  Detail & Related papers  (2024-09-29T04:31:45Z)
• Six Fallacies in Substituting Large Language Models for Human Participants [0.0]
  Can AI systems like large language models (LLMs) replace human participants in behavioral and psychological research?<n>Here I critically evaluate the "replacement" perspective and identify six interpretive fallacies that undermine its validity.<n>Each fallacy represents a potential misunderstanding about what LLMs are and what they can tell us about human cognition.
  arXiv  Detail & Related papers  (2024-02-06T23:28:23Z)
• Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision [84.31474052176343]
  Recent AI-assistant agents, such as ChatGPT, rely on supervised fine-tuning (SFT) with human annotations and reinforcement learning from human feedback to align the output with human intentions. This dependence can significantly constrain the true potential of AI-assistant agents due to the high cost of obtaining human supervision. We propose a novel approach called SELF-ALIGN, which combines principle-driven reasoning and the generative power of LLMs for the self-alignment of AI agents with minimal human supervision.
  arXiv  Detail & Related papers  (2023-05-04T17:59:28Z)
• Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making [53.62514158534574]
  We study whether features that reveal case-specific model information can calibrate trust and improve the joint performance of the human and AI. We show that confidence score can help calibrate people's trust in an AI model, but trust calibration alone is not sufficient to improve AI-assisted decision making.
  arXiv  Detail & Related papers  (2020-01-07T15:33:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.