Comparing Machines and Children: Using Developmental Psychology
Experiments to Assess the Strengths and Weaknesses of LaMDA Responses
- URL: http://arxiv.org/abs/2305.11243v2
- Date: Tue, 7 Nov 2023 20:20:47 GMT
- Title: Comparing Machines and Children: Using Developmental Psychology
Experiments to Assess the Strengths and Weaknesses of LaMDA Responses
- Authors: Eliza Kosoy, Emily Rose Reagan, Leslie Lai, Alison Gopnik and Danielle
Krettek Cobb
- Abstract summary: We adapt classical developmental experiments to evaluate the capabilities of LaMDA, a large language model from Google.
We find that LaMDA generates appropriate responses similar to those of children in experiments involving social understanding.
On the other hand, LaMDA's responses in early object and action understanding, theory of mind, and especially causal reasoning tasks are very different from those of young children.
- Score: 0.02999888908665658
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Developmental psychologists have spent decades devising experiments to test
the intelligence and knowledge of infants and children, tracing the origin of
crucial concepts and capacities. Moreover, experimental techniques in
developmental psychology have been carefully designed to discriminate the
cognitive capacities that underlie particular behaviors. We propose that using
classical experiments from child development is a particularly effective way to
probe the computational abilities of AI models, in general, and LLMs in
particular. First, the methodological techniques of developmental psychology,
such as the use of novel stimuli to control for past experience or control
conditions to determine whether children are using simple associations, can be
equally helpful for assessing the capacities of LLMs. In parallel, testing LLMs
in this way can tell us whether the information that is encoded in text is
sufficient to enable particular responses, or whether those responses depend on
other kinds of information, such as information from exploration of the
physical world. In this work we adapt classical developmental experiments to
evaluate the capabilities of LaMDA, a large language model from Google. We
propose a novel LLM Response Score (LRS) metric which can be used to evaluate
other language models, such as GPT. We find that LaMDA generates appropriate
responses that are similar to those of children in experiments involving social
understanding, perhaps providing evidence that knowledge of these domains is
discovered through language. On the other hand, LaMDA's responses in early
object and action understanding, theory of mind, and especially causal
reasoning tasks are very different from those of young children, perhaps
showing that these domains require more real-world, self-initiated exploration
and cannot simply be learned from patterns in language input.
Related papers
- Development of Cognitive Intelligence in Pre-trained Language Models [3.1815791977708834]
Recent studies show evidence for emergent cognitive abilities in Large Pre-trained Language Models.
The developmental trajectories of PLMs consistently exhibit a window of maximal alignment to human cognitive development.
After that window, training appears to serve the engineering goal of reducing loss but not the scientific goal of increasing alignment with human cognition.
arXiv Detail & Related papers (2024-07-01T07:56:36Z) - Exploring Spatial Schema Intuitions in Large Language and Vision Models [8.944921398608063]
We investigate whether large language models (LLMs) effectively capture implicit human intuitions about building blocks of language.
Surprisingly, correlations between model outputs and human responses emerge, revealing adaptability without a tangible connection to embodied experiences.
This research contributes to a nuanced understanding of the interplay between language, spatial experiences, and computations made by large language models.
arXiv Detail & Related papers (2024-02-01T19:25:50Z) - Exploring the Frontiers of LLMs in Psychological Applications: A Comprehensive Review [4.147674289030404]
Large language models (LLMs) have the potential to simulate aspects of human cognition and behavior.
LLMs offer innovative tools for literature review, hypothesis generation, experimental design, experimental subjects, data analysis, academic writing, and peer review in psychology.
There are issues like data privacy, the ethical implications of using LLMs in psychological research, and the need for a deeper understanding of these models' limitations.
arXiv Detail & Related papers (2024-01-03T03:01:29Z) - Divergences between Language Models and Human Brains [63.405788999891335]
Recent research has hinted that brain signals can be effectively predicted using internal representations of language models (LMs)
We show that there are clear differences in how LMs and humans represent and use language.
We identify two domains that are not captured well by LMs: social/emotional intelligence and physical commonsense.
arXiv Detail & Related papers (2023-11-15T19:02:40Z) - Character-LLM: A Trainable Agent for Role-Playing [67.35139167985008]
Large language models (LLMs) can be used to serve as agents to simulate human behaviors.
We introduce Character-LLM that teach LLMs to act as specific people such as Beethoven, Queen Cleopatra, Julius Caesar, etc.
arXiv Detail & Related papers (2023-10-16T07:58:56Z) - Exploring the Cognitive Knowledge Structure of Large Language Models: An
Educational Diagnostic Assessment Approach [50.125704610228254]
Large Language Models (LLMs) have not only exhibited exceptional performance across various tasks, but also demonstrated sparks of intelligence.
Recent studies have focused on assessing their capabilities on human exams and revealed their impressive competence in different domains.
We conduct an evaluation using MoocRadar, a meticulously annotated human test dataset based on Bloom taxonomy.
arXiv Detail & Related papers (2023-10-12T09:55:45Z) - Machine Psychology: Investigating Emergent Capabilities and Behavior in Large Language Models Using Psychological Methods [0.0]
Large language models (LLMs) are currently at the forefront of intertwining AI systems with human communication and everyday life.
This paper introduces a new field of research called "machine psychology"
It defines methodological standards for machine psychology research, especially by focusing on policies for prompt designs.
arXiv Detail & Related papers (2023-03-24T13:24:41Z) - Evaluating and Inducing Personality in Pre-trained Language Models [78.19379997967191]
We draw inspiration from psychometric studies by leveraging human personality theory as a tool for studying machine behaviors.
To answer these questions, we introduce the Machine Personality Inventory (MPI) tool for studying machine behaviors.
MPI follows standardized personality tests, built upon the Big Five Personality Factors (Big Five) theory and personality assessment inventories.
We devise a Personality Prompting (P2) method to induce LLMs with specific personalities in a controllable way.
arXiv Detail & Related papers (2022-05-20T07:32:57Z) - Is the Computation of Abstract Sameness Relations Human-Like in Neural
Language Models? [4.0810783261728565]
This work explores whether state-of-the-art NLP models exhibit elementary mechanisms known from human cognition.
The computation of "abstract sameness relations" is assumed to play an important role in human language acquisition and processing.
arXiv Detail & Related papers (2022-05-12T15:19:54Z) - From Psychological Curiosity to Artificial Curiosity: Curiosity-Driven
Learning in Artificial Intelligence Tasks [56.20123080771364]
Psychological curiosity plays a significant role in human intelligence to enhance learning through exploration and information acquisition.
In the Artificial Intelligence (AI) community, artificial curiosity provides a natural intrinsic motivation for efficient learning.
CDL has become increasingly popular, where agents are self-motivated to learn novel knowledge.
arXiv Detail & Related papers (2022-01-20T17:07:03Z) - AGENT: A Benchmark for Core Psychological Reasoning [60.35621718321559]
Intuitive psychology is the ability to reason about hidden mental variables that drive observable actions.
Despite recent interest in machine agents that reason about other agents, it is not clear if such agents learn or hold the core psychology principles that drive human reasoning.
We present a benchmark consisting of procedurally generated 3D animations, AGENT, structured around four scenarios.
arXiv Detail & Related papers (2021-02-24T14:58:23Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.