Do Large Language Models Show Decision Heuristics Similar to Humans? A
Case Study Using GPT-3.5
- URL: http://arxiv.org/abs/2305.04400v1
- Date: Mon, 8 May 2023 01:02:52 GMT
- Title: Do Large Language Models Show Decision Heuristics Similar to Humans? A
Case Study Using GPT-3.5
- Authors: Gaurav Suri, Lily R. Slater, Ali Ziaee, Morgan Nguyen
- Abstract summary: GPT-3.5 is an example of an LLM that supports a conversational agent called ChatGPT.
In this work, we used a series of novel prompts to determine whether ChatGPT shows biases, and other decision effects.
We also tested the same prompts on human participants.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: A Large Language Model (LLM) is an artificial intelligence system that has
been trained on vast amounts of natural language data, enabling it to generate
human-like responses to written or spoken language input. GPT-3.5 is an example
of an LLM that supports a conversational agent called ChatGPT. In this work, we
used a series of novel prompts to determine whether ChatGPT shows heuristics,
biases, and other decision effects. We also tested the same prompts on human
participants. Across four studies, we found that ChatGPT was influenced by
random anchors in making estimates (Anchoring Heuristic, Study 1); it judged
the likelihood of two events occurring together to be higher than the
likelihood of either event occurring alone, and it was erroneously influenced
by salient anecdotal information (Representativeness and Availability
Heuristic, Study 2); it found an item to be more efficacious when its features
were presented positively rather than negatively - even though both
presentations contained identical information (Framing Effect, Study 3); and it
valued an owned item more than a newly found item even though the two items
were identical (Endowment Effect, Study 4). In each study, human participants
showed similar effects. Heuristics and related decision effects in humans are
thought to be driven by cognitive and affective processes such as loss aversion
and effort reduction. The fact that an LLM - which lacks these processes - also
shows such effects invites consideration of the possibility that language may
play a role in generating these effects in humans.
Related papers
- Humanity in AI: Detecting the Personality of Large Language Models [0.0]
Questionnaires are a common method for detecting the personality of Large Language Models (LLMs)
We propose combining text mining with questionnaires method.
We find that the personalities of LLMs are derived from their pre-trained data.
arXiv Detail & Related papers (2024-10-11T05:53:11Z) - Cross-lingual Speech Emotion Recognition: Humans vs. Self-Supervised Models [16.0617753653454]
This study presents a comparative analysis between human performance and SSL models.
We also compare the SER ability of models and humans at both utterance- and segment-levels.
Our findings reveal that models, with appropriate knowledge transfer, can adapt to the target language and achieve performance comparable to native speakers.
arXiv Detail & Related papers (2024-09-25T13:27:17Z) - Rel-A.I.: An Interaction-Centered Approach To Measuring Human-LM Reliance [73.19687314438133]
We study how reliance is affected by contextual features of an interaction.
We find that contextual characteristics significantly affect human reliance behavior.
Our results show that calibration and language quality alone are insufficient in evaluating the risks of human-LM interactions.
arXiv Detail & Related papers (2024-07-10T18:00:05Z) - Towards a Psychology of Machines: Large Language Models Predict Human Memory [0.0]
Large language models (LLMs) have shown remarkable abilities in natural language processing.
This study explores whether LLMs can predict human memory performance in tasks involving garden-path sentences and contextual information.
arXiv Detail & Related papers (2024-03-08T08:41:14Z) - Divergences between Language Models and Human Brains [59.100552839650774]
We systematically explore the divergences between human and machine language processing.
We identify two domains that LMs do not capture well: social/emotional intelligence and physical commonsense.
Our results show that fine-tuning LMs on these domains can improve their alignment with human brain responses.
arXiv Detail & Related papers (2023-11-15T19:02:40Z) - Sensitivity, Performance, Robustness: Deconstructing the Effect of
Sociodemographic Prompting [64.80538055623842]
sociodemographic prompting is a technique that steers the output of prompt-based models towards answers that humans with specific sociodemographic profiles would give.
We show that sociodemographic information affects model predictions and can be beneficial for improving zero-shot learning in subjective NLP tasks.
arXiv Detail & Related papers (2023-09-13T15:42:06Z) - Cognitive Effects in Large Language Models [14.808777775761753]
Large Language Models (LLMs) have received enormous attention over the past year and are now used by hundreds of millions of people every day.
We tested one of these models (GPT-3) on a range of cognitive effects, which are systematic patterns that are usually found in human cognitive tasks.
Specifically, we show that the priming, distance, SNARC, and size congruity effects were presented with GPT-3, while the anchoring effect is absent.
arXiv Detail & Related papers (2023-08-28T06:30:33Z) - Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias [57.42417061979399]
Recent studies show that instruction tuning (IT) and reinforcement learning from human feedback (RLHF) improve the abilities of large language models (LMs) dramatically.
In this work, we investigate the effect of IT and RLHF on decision making and reasoning in LMs.
Our findings highlight the presence of these biases in various models from the GPT-3, Mistral, and T5 families.
arXiv Detail & Related papers (2023-08-01T01:39:25Z) - Susceptibility to Influence of Large Language Models [5.931099001882958]
Two studies tested the hypothesis that a Large Language Model (LLM) can be used to model psychological change following exposure to influential input.
The first study tested a generic mode of influence - the Illusory Truth Effect (ITE)
The second study concerns a specific mode of influence - populist framing of news to increase its persuasion and political mobilization.
arXiv Detail & Related papers (2023-03-10T16:53:30Z) - Naturalistic Causal Probing for Morpho-Syntax [76.83735391276547]
We suggest a naturalistic strategy for input-level intervention on real world data in Spanish.
Using our approach, we isolate morpho-syntactic features from counfounders in sentences.
We apply this methodology to analyze causal effects of gender and number on contextualized representations extracted from pre-trained models.
arXiv Detail & Related papers (2022-05-14T11:47:58Z) - Model-based analysis of brain activity reveals the hierarchy of language
in 305 subjects [82.81964713263483]
A popular approach to decompose the neural bases of language consists in correlating, across individuals, the brain responses to different stimuli.
Here, we show that a model-based approach can reach equivalent results within subjects exposed to natural stimuli.
arXiv Detail & Related papers (2021-10-12T15:30:21Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.