Can Humans Tell? A Dual-Axis Study of Human Perception of LLM-Generated News

• URL: http://arxiv.org/abs/2604.03755v1
• Date: Sat, 04 Apr 2026 15:03:42 GMT
• Title: Can Humans Tell? A Dual-Axis Study of Human Perception of LLM-Generated News
• Abstract summary: Can humans tell whether a news article was written by a person or a large language model (LLM)?<n>We investigate this question using JudgeGPT, a study platform that measures source attribution (human vs. machine) and authenticity judgment (legitimate vs. fake) on continuous scales.<n>We report five findings: (1) participants cannot reliably distinguish machine-generated from human-written text; (2) this inability holds across all tested models, including open-weight models with as few as 7B parameters; and (3) accuracy degrades after approximately 30 sequential evaluations due to cognitive fatigue.
• Score: 47.03825808787752
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Can humans tell whether a news article was written by a person or a large language model (LLM)? We investigate this question using JudgeGPT, a study platform that independently measures source attribution (human vs. machine) and authenticity judgment (legitimate vs. fake) on continuous scales. From 2,318 judgments collected from 1,054 participants across content generated by six LLMs, we report five findings: (1) participants cannot reliably distinguish machine-generated from human-written text (p > .05, Welch's t-test); (2) this inability holds across all tested models, including open-weight models with as few as 7B parameters; (3) self-reported domain expertise predicts judgment accuracy (r = .35, p < .001) whereas political orientation does not (r = -.10, n.s.); (4) clustering reveals distinct response strategies ("Skeptics" vs. "Believers"); and (5) accuracy degrades after approximately 30 sequential evaluations due to cognitive fatigue. The answer, in short, is no: humans cannot reliably tell. These results indicate that user-side detection is not a viable defense and motivate system-level countermeasures such as cryptographic content provenance.

Related papers

• NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment [54.4265627247104]
  Large language models (LLMs) are increasingly used in peer review at major AI conferences.<n>Existing benchmarks assess novelty as a single holistic score.<n>We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis.
  arXiv  Detail & Related papers  (2026-09-10T08:34:56Z)
• Interrupting the Chain: Human Perception of AI-Generated Disinformation Through a Kill Chain Lens [47.03825808787752]
  We present empirical findings from a human-subject study in which users classified news fragments by origin (human vs. machine) and veracity (real vs. fake)<n>We organize results using an adapted cybersecurity kill chain as a taxonomy for intervention, mapping perception data onto stages of a cognitive attack lifecycle.<n>Three key findings emerge: (1) a perception-accuracy gap where heightened suspicion does not improve detection; (2) modern LLMs frequently produce human-indistinguishable text; and (3) an asymmetric cognitive fatigue effect where fake-news detection degrades by 10.2 percentage points under sustained exposure while AI-origin detection remains stable.
  arXiv  Detail & Related papers  (2026-07-26T15:40:41Z)
• Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality? [74.69723255239663]
  Grounded Personality Reasoning requires MLLMs to anchor each Big Five rating in observable evidence through a chain of rating, reasoning, and grounding.<n> MM-OCEAN is produced by a multi-agent pipeline with human verification, with timestamped behavioral observations, evidence-grounded trait analyses, and seven categories of cue-grounding MCQs.<n>The analysis uncovers a striking Prejudice Gap: across the field, 51% of correct ratings are not grounded in retrieved cues, and the Holistic-Grounding Rate spans only 0-33.5%.
  arXiv  Detail & Related papers  (2026-05-21T07:42:47Z)
• Sem-Detect: Semantic Level Detection of AI Generated Peer-Reviews [11.21278193949403]
  We propose Sem-Detect, an authorship detection method for peer reviews.<n>It compares a target review against multiple AI-generated reviews of the same paper.<n>It is able to distinguish fully AI reviews from authentic human-written ones.
  arXiv  Detail & Related papers  (2026-05-20T20:19:16Z)
• GPT4o-Receipt: A Dataset and Human Study for AI-Generated Document Forensics [12.448893104517808]
  We present GPT4o-Receipt, a benchmark of 1,235 receipt images pairing GPT-4o-generated receipts with authentic ones from established datasets.<n>Human annotators exhibit the largest visual discrimination gap of any evaluator, yet their binary detection F1 falls well below Claude Sonnet 4 and below Gemini 2.5 Flash.<n>Our five-model evaluation reveals dramatic performance disparities and calibration differences that render simple accuracy metrics insufficient for detector selection.
  arXiv  Detail & Related papers  (2026-03-12T02:05:27Z)
• Eroding the Truth-Default: A Causal Analysis of Human Susceptibility to Foundation Model Hallucinations and Disinformation in the Wild [47.03825808787752]
  "Fake news familiarity" emerges as a candidate mediator, suggesting that exposure may function as adversarial training for human discriminators.<n>These findings suggest that "pre-bunking" interventions should target cognitive source monitoring rather than demographic segmentation.
  arXiv  Detail & Related papers  (2026-01-30T11:49:58Z)
• Large Language Models Pass the Turing Test [0.913127392774573]
  We evaluated 4 systems (ELIZA, GPT-4o, LLaMa-3.1-405B, and GPT-4.5) in two Turing tests on independent populations.<n>Results constitute the first empirical evidence that any artificial system passes a standard three-party Turing test.
  arXiv  Detail & Related papers  (2025-03-31T02:37:45Z)
• ExaGPT: Example-Based Machine-Generated Text Detection for Human Interpretability [62.285407189502216]
  Detecting texts generated by Large Language Models (LLMs) could cause grave mistakes due to incorrect decisions.<n>We introduce ExaGPT, an interpretable detection approach grounded in the human decision-making process.<n>We show that ExaGPT massively outperforms prior powerful detectors by up to +40.9 points of accuracy at a false positive rate of 1%.
  arXiv  Detail & Related papers  (2025-02-17T01:15:07Z)
• "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust [51.542856739181474]
  We show how different natural language expressions of uncertainty impact participants' reliance, trust, and overall task performance. We find that first-person expressions decrease participants' confidence in the system and tendency to agree with the system's answers, while increasing participants' accuracy. Our findings suggest that using natural language expressions of uncertainty may be an effective approach for reducing overreliance on LLMs, but that the precise language used matters.
  arXiv  Detail & Related papers  (2024-05-01T16:43:55Z)
• How Well Can LLMs Echo Us? Evaluating AI Chatbots' Role-Play Ability with ECHO [55.25989137825992]
  We introduce ECHO, an evaluative framework inspired by the Turing test. This framework engages the acquaintances of the target individuals to distinguish between human and machine-generated responses. We evaluate three role-playing LLMs using ECHO, with GPT-3.5 and GPT-4 serving as foundational models.
  arXiv  Detail & Related papers  (2024-04-22T08:00:51Z)
• Six Fallacies in Substituting Large Language Models for Human Participants [0.0]
  Can AI systems like large language models (LLMs) replace human participants in behavioral and psychological research?<n>Here I critically evaluate the "replacement" perspective and identify six interpretive fallacies that undermine its validity.<n>Each fallacy represents a potential misunderstanding about what LLMs are and what they can tell us about human cognition.
  arXiv  Detail & Related papers  (2024-02-06T23:28:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.