Stronger Language Models Produce More Human-Like Errors

• URL: http://arxiv.org/abs/2506.11128v1
• Date: Tue, 10 Jun 2025 17:04:33 GMT
• Title: Stronger Language Models Produce More Human-Like Errors
• Authors: Andrew Keenan Richardson, Ryan Othniel Kearns, Sean Moss, Vincent Wang-Mascianica, Philipp Koralus,
• Abstract summary: We show that while overall reasoning capabilities increase with model sophistication, the nature of errors increasingly mirrors predictable human reasoning fallacies.<n>We generate logical reasoning problems where humans predictably err, evaluating 38 language models across 383 reasoning tasks.<n>Our analysis indicates that as models advance in general capability, the proportion of their incorrect answers that align with ETR-predicted human fallacies tends to increase.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Do language models converge toward human-like reasoning patterns as they improve? We provide surprising evidence that while overall reasoning capabilities increase with model sophistication, the nature of errors increasingly mirrors predictable human reasoning fallacies: a previously unobserved inverse scaling phenomenon. To investigate this question, we apply the Erotetic Theory of Reasoning (ETR), a formal cognitive framework with empirical support for predicting human reasoning outcomes. Using the open-source package PyETR, we generate logical reasoning problems where humans predictably err, evaluating responses from 38 language models across 383 reasoning tasks. Our analysis indicates that as models advance in general capability (as measured by Chatbot Arena scores), the proportion of their incorrect answers that align with ETR-predicted human fallacies tends to increase ($\rho = 0.360, p = 0.0265$). Notably, as we observe no correlation between model sophistication and logical correctness on these tasks, this shift in error patterns toward human-likeness occurs independently of error rate. These findings challenge the prevailing view that scaling language models naturally obtains normative rationality, suggesting instead a convergence toward human-like cognition inclusive of our characteristic biases and limitations, as we further confirm by demonstrating order-effects in language model reasoning.

Related papers

• CLEAR-3K: Assessing Causal Explanatory Capabilities in Language Models [3.137688620241855]
  We introduce CLEAR-3K, a dataset of 3,000 assertion-reasoning questions designed to evaluate whether language models can determine if one statement causally explains another.<n>Each question present an assertion-reason pair and challenge language models to distinguish between semantic relatedness and genuine causal explanatory relationships.
  arXiv  Detail & Related papers  (2025-06-20T17:35:36Z)
• A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Models [53.18562650350898]
  Chain-of-thought (CoT) reasoning enhances performance of large language models.<n>We present the first comprehensive study of CoT faithfulness in large vision-language models.
  arXiv  Detail & Related papers  (2025-05-29T18:55:05Z)
• Evaluating the Logical Reasoning Abilities of Large Reasoning Models [15.009205651973666]
  We introduce LogiEval, a benchmark for evaluating logical reasoning in large reasoning models.<n>LogiEval spans diverse reasoning types (deductive, inductive, analogical, and abductive) and task formats (e.g., logical sequence, argument analysis)<n>Our experiments demonstrate that modern reasoning models excel at 4-choice argument analysis problems and analogical reasoning, surpassing human performance.<n>Our analysis reveals that human performance does not mirror model failure distributions.
  arXiv  Detail & Related papers  (2025-05-17T05:36:14Z)
• Can Language Models Learn to Skip Steps? [59.84848399905409]
  We study the ability to skip steps in reasoning. Unlike humans, who may skip steps to enhance efficiency or to reduce cognitive load, models do not possess such motivations. Our work presents the first exploration into human-like step-skipping ability.
  arXiv  Detail & Related papers  (2024-11-04T07:10:24Z)
• UNcommonsense Reasoning: Abductive Reasoning about Uncommon Situations [62.71847873326847]
  We investigate the ability to model unusual, unexpected, and unlikely situations. Given a piece of context with an unexpected outcome, this task requires reasoning abductively to generate an explanation. We release a new English language corpus called UNcommonsense.
  arXiv  Detail & Related papers  (2023-11-14T19:00:55Z)
• A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models [39.77445889769015]
  We show that, within the PaLM2 family of language models, larger models are more logical than smaller ones. Even the largest models make systematic errors, some of which mirror human reasoning biases. Overall, we find that language models often mimic the human biases included in their training data, but are able to overcome them in some cases.
  arXiv  Detail & Related papers  (2023-11-01T11:13:06Z)
• The Generative AI Paradox: "What It Can Create, It May Not Understand" [81.89252713236746]
  Recent wave of generative AI has sparked excitement and concern over potentially superhuman levels of artificial intelligence. At the same time, models still show basic errors in understanding that would not be expected even in non-expert humans. This presents us with an apparent paradox: how do we reconcile seemingly superhuman capabilities with the persistence of errors that few humans would make?
  arXiv  Detail & Related papers  (2023-10-31T18:07:07Z)
• A fine-grained comparison of pragmatic language understanding in humans and language models [2.231167375820083]
  We compare language models and humans on seven pragmatic phenomena. We find that the largest models achieve high accuracy and match human error patterns. Preliminary evidence that models and humans are sensitive to similar linguistic cues.
  arXiv  Detail & Related papers  (2022-12-13T18:34:59Z)
• To what extent do human explanations of model behavior align with actual model behavior? [91.67905128825402]
  We investigated the extent to which human-generated explanations of models' inference decisions align with how models actually make these decisions. We defined two alignment metrics that quantify how well natural language human explanations align with model sensitivity to input words. We find that a model's alignment with human explanations is not predicted by the model's accuracy on NLI.
  arXiv  Detail & Related papers  (2020-12-24T17:40:06Z)
• Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction [49.254162397086006]
  We study explanations based on visual saliency in an image-based age prediction task. We find that presenting model predictions improves human accuracy. However, explanations of various kinds fail to significantly alter human accuracy or trust in the model.
  arXiv  Detail & Related papers  (2020-07-23T20:39:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.