Implicit Causality-biases in humans and LLMs as a tool for benchmarking LLM discourse capabilities

• URL: http://arxiv.org/abs/2501.12980v1
• Date: Wed, 22 Jan 2025 16:07:24 GMT
• Title: Implicit Causality-biases in humans and LLMs as a tool for benchmarking LLM discourse capabilities
• Authors: Florian Kankowski, Torgrim Solstad, Sina Zarriess, Oliver Bott,
• Abstract summary: We compare data generated with mono- and multilingual LLMs spanning a range of model sizes with data provided by human participants.<n>We aim to develop a benchmark to assess the capabilities of LLMs with discourse biases as a robust proxy for more general discourse understanding capabilities.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In this paper, we compare data generated with mono- and multilingual LLMs spanning a range of model sizes with data provided by human participants in an experimental setting investigating well-established discourse biases. Beyond the comparison as such, we aim to develop a benchmark to assess the capabilities of LLMs with discourse biases as a robust proxy for more general discourse understanding capabilities. More specifically, we investigated Implicit Causality verbs, for which psycholinguistic research has found participants to display biases with regard to three phenomena:\ the establishment of (i) coreference relations (Experiment 1), (ii) coherence relations (Experiment 2), and (iii) the use of particular referring expressions (Experiments 3 and 4). With regard to coreference biases we found only the largest monolingual LLM (German Bloom 6.4B) to display more human-like biases. For coherence relation, no LLM displayed the explanation bias usually found for humans. For referring expressions, all LLMs displayed a preference for referring to subject arguments with simpler forms than to objects. However, no bias effect on referring expression was found, as opposed to recent studies investigating human biases.

Related papers

• Evaluating how LLM annotations represent diverse views on contentious topics [3.405231040967506]
  We show how generative large language models (LLMs) represent diverse viewpoints on contentious labeling tasks. Our findings suggest that when using LLMs to annotate data, under-representing the views of particular groups is not a substantial concern.
  arXiv  Detail & Related papers  (2025-03-29T22:53:15Z)
• Implicit Bias in LLMs: A Survey [2.07180164747172]
  This paper provides a comprehensive review of the existing literature on implicit bias in Large language models. We begin by introducing key concepts, theories and methods related to implicit bias in psychology. We categorize detection methods into three primary approaches: word association, task-oriented text generation and decision-making.
  arXiv  Detail & Related papers  (2025-03-04T16:49:37Z)
• ExpliCa: Evaluating Explicit Causal Reasoning in Large Language Models [75.05436691700572]
  We introduce ExpliCa, a new dataset for evaluating Large Language Models (LLMs) in explicit causal reasoning. We tested seven commercial and open-source LLMs on ExpliCa through prompting and perplexity-based metrics. Surprisingly, models tend to confound temporal relations with causal ones, and their performance is also strongly influenced by the linguistic order of the events.
  arXiv  Detail & Related papers  (2025-02-21T14:23:14Z)
• Anchoring Bias in Large Language Models: An Experimental Study [5.229564709919574]
  Large Language Models (LLMs) like GPT-4 and Gemini have significantly advanced artificial intelligence.<n>This study delves into anchoring bias, a cognitive bias where initial information disproportionately influences judgment.
  arXiv  Detail & Related papers  (2024-12-09T15:45:03Z)
• Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective [66.34066553400108]
  We conduct a rigorous evaluation of large language models' implicit bias towards certain demographics. Inspired by psychometric principles, we propose three attack approaches, i.e., Disguise, Deception, and Teaching. Our methods can elicit LLMs' inner bias more effectively than competitive baselines.
  arXiv  Detail & Related papers  (2024-06-20T06:42:08Z)
• White Men Lead, Black Women Help? Benchmarking Language Agency Social Biases in LLMs [58.27353205269664]
  Social biases can manifest in language agency. We introduce the novel Language Agency Bias Evaluation benchmark. We unveil language agency social biases in 3 recent Large Language Model (LLM)-generated content.
  arXiv  Detail & Related papers  (2024-04-16T12:27:54Z)
• Exploring Value Biases: How LLMs Deviate Towards the Ideal [57.99044181599786]
  Large-Language-Models (LLMs) are deployed in a wide range of applications, and their response has an increasing social impact. We show that value bias is strong in LLMs across different categories, similar to the results found in human studies.
  arXiv  Detail & Related papers  (2024-02-16T18:28:43Z)
• Exploring the Jungle of Bias: Political Bias Attribution in Language Models via Dependency Analysis [86.49858739347412]
  Large Language Models (LLMs) have sparked intense debate regarding the prevalence of bias in these models and its mitigation. We propose a prompt-based method for the extraction of confounding and mediating attributes which contribute to the decision process. We find that the observed disparate treatment can at least in part be attributed to confounding and mitigating attributes and model misalignment.
  arXiv  Detail & Related papers  (2023-11-15T00:02:25Z)
• "Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters [97.11173801187816]
  Large Language Models (LLMs) have recently emerged as an effective tool to assist individuals in writing various types of content. This paper critically examines gender biases in LLM-generated reference letters.
  arXiv  Detail & Related papers  (2023-10-13T16:12:57Z)
• Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds [59.71218039095155]
  We evaluate language understanding capacities on simple inference tasks that most humans find trivial. We target (i) grammatically-specified entailments, (ii) premises with evidential adverbs of uncertainty, and (iii) monotonicity entailments. The models exhibit moderate to low performance on these evaluation sets.
  arXiv  Detail & Related papers  (2023-05-24T06:41:09Z)
• Sources of Hallucination by Large Language Models on Inference Tasks [16.644096408742325]
  Large Language Models (LLMs) are claimed to be capable of Natural Language Inference (NLI) We present a series of behavioral studies on several LLM families which probe their behavior using controlled experiments.
  arXiv  Detail & Related papers  (2023-05-23T22:24:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.