Using cognitive psychology to understand GPT-3

• URL: http://arxiv.org/abs/2206.14576v1
• Date: Tue, 21 Jun 2022 20:06:03 GMT
• Title: Using cognitive psychology to understand GPT-3
• Authors: Marcel Binz and Eric Schulz
• Abstract summary: We study GPT-3, a recent large language model, using tools from cognitive psychology. We assess GPT-3's decision-making, information search, deliberation, and causal reasoning abilities.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study GPT-3, a recent large language model, using tools from cognitive psychology. More specifically, we assess GPT-3's decision-making, information search, deliberation, and causal reasoning abilities on a battery of canonical experiments from the literature. We find that much of GPT-3's behavior is impressive: it solves vignette-based tasks similarly or better than human subjects, is able to make decent decisions from descriptions, outperforms humans in a multi-armed bandit task, and shows signatures of model-based reinforcement learning. Yet we also find that small perturbations to vignette-based tasks can lead GPT-3 vastly astray, that it shows no signatures of directed exploration, and that it fails miserably in a causal reasoning task. These results enrich our understanding of current large language models and pave the way for future investigations using tools from cognitive psychology to study increasingly capable and opaque artificial agents.

Related papers

• CausalGym: Benchmarking causal interpretability methods on linguistic tasks [52.61917615039112]
  We use CausalGym to benchmark the ability of interpretability methods to causally affect model behaviour. We study the pythia models (14M--6.9B) and assess the causal efficacy of a wide range of interpretability methods. We find that DAS outperforms the other methods, and so we use it to study the learning trajectory of two difficult linguistic phenomena.
  arXiv  Detail & Related papers  (2024-02-19T21:35:56Z)
• Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias [57.42417061979399]
  Recent studies show that instruction tuning (IT) and reinforcement learning from human feedback (RLHF) improve the abilities of large language models (LMs) dramatically. In this work, we investigate the effect of IT and RLHF on decision making and reasoning in LMs. Our findings highlight the presence of these biases in various models from the GPT-3, Mistral, and T5 families.
  arXiv  Detail & Related papers  (2023-08-01T01:39:25Z)
• Human-Like Intuitive Behavior and Reasoning Biases Emerged in Language Models -- and Disappeared in GPT-4 [0.0]
  We show that large language models (LLMs) exhibit behavior that resembles human-like intuition. We also probe how sturdy the inclination for intuitive-like decision-making is.
  arXiv  Detail & Related papers  (2023-06-13T08:43:13Z)
• Systematic Evaluation of GPT-3 for Zero-Shot Personality Estimation [12.777659013330823]
  GPT-3 is used to estimate the Big 5 personality traits from users' social media posts. We find that GPT-3 performance is somewhat close to an existing pre-trained SotA for broad classification. We analyze where GPT-3 performs better, as well as worse, than a pretrained lexical model, illustrating systematic errors.
  arXiv  Detail & Related papers  (2023-06-01T22:43:37Z)
• Let GPT be a Math Tutor: Teaching Math Word Problem Solvers with Customized Exercise Generation [39.282695549919495]
  We present a novel approach for distilling math word problem solving capabilities from large language models (LLMs) into smaller, more efficient student models. Our approach is designed to consider the student model's weaknesses and foster a tailored learning experience by generating targeted exercises aligned with educational science principles.
  arXiv  Detail & Related papers  (2023-05-22T17:36:14Z)
• Mind meets machine: Unravelling GPT-4's cognitive psychology [0.7302002320865727]
  Large language models (LLMs) are emerging as potent tools increasingly capable of performing human-level tasks. This study focuses on the evaluation of GPT-4's performance on datasets such as CommonsenseQA, SuperGLUE, MATH and HANS. We show that GPT-4 exhibits a high level of accuracy in cognitive psychology tasks relative to the prior state-of-the-art models.
  arXiv  Detail & Related papers  (2023-03-20T20:28:26Z)
• Evaluating Psychological Safety of Large Language Models [72.88260608425949]
  We designed unbiased prompts to evaluate the psychological safety of large language models (LLMs) We tested five different LLMs by using two personality tests: Short Dark Triad (SD-3) and Big Five Inventory (BFI) Despite being instruction fine-tuned with safety metrics to reduce toxicity, InstructGPT, GPT-3.5, and GPT-4 still showed dark personality patterns. Fine-tuning Llama-2-chat-7B with responses from BFI using direct preference optimization could effectively reduce the psychological toxicity of the model.
  arXiv  Detail & Related papers  (2022-12-20T18:45:07Z)
• Emergent Analogical Reasoning in Large Language Models [1.5469452301122177]
  We show that GPT-3 has a surprisingly strong capacity for abstract pattern induction, matching or even surpassing human capabilities in most settings. Our results indicate that large language models such as GPT-3 have acquired an emergent ability to find zero-shot solutions to a broad range of analogy problems.
  arXiv  Detail & Related papers  (2022-12-19T00:04:56Z)
• Elaboration-Generating Commonsense Question Answering at Scale [77.96137534751445]
  In question answering requiring common sense, language models (e.g., GPT-3) have been used to generate text expressing background knowledge. We finetune smaller language models to generate useful intermediate context, referred to here as elaborations. Our framework alternates between updating two language models -- an elaboration generator and an answer predictor -- allowing each to influence the other.
  arXiv  Detail & Related papers  (2022-09-02T18:32:09Z)
• The Unreliability of Explanations in Few-Shot In-Context Learning [50.77996380021221]
  We focus on two NLP tasks that involve reasoning over text, namely question answering and natural language inference. We show that explanations judged as good by humans--those that are logically consistent with the input--usually indicate more accurate predictions. We present a framework for calibrating model predictions based on the reliability of the explanations.
  arXiv  Detail & Related papers  (2022-05-06T17:57:58Z)
• Language Models are Few-Shot Learners [61.36677350504291]
  We show that scaling up language models greatly improves task-agnostic, few-shot performance. We train GPT-3, an autoregressive language model with 175 billion parameters, and test its performance in the few-shot setting. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks.
  arXiv  Detail & Related papers  (2020-05-28T17:29:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.