Do large language models solve verbal analogies like children do?

• URL: http://arxiv.org/abs/2310.20384v1
• Date: Tue, 31 Oct 2023 11:49:11 GMT
• Title: Do large language models solve verbal analogies like children do?
• Authors: Claire E. Stevenson, Mathilde ter Veen, Rochelle Choenni, Han L. J. van der Maas and Ekaterina Shutova
• Abstract summary: This paper investigates whether large language models (LLMs) solve verbal analogies in A:B::C:? form using associations, similar to what children do. We use verbal analogies extracted from an online adaptive learning environment, where 14,002 7-12 year-olds from the Netherlands solved 622 analogies in Dutch. We conclude that the LLMs we tested indeed tend to solve verbal analogies by association with C like children do.
• Score: 10.616401727158626
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Analogy-making lies at the heart of human cognition. Adults solve analogies such as \textit{Horse belongs to stable like chicken belongs to ...?} by mapping relations (\textit{kept in}) and answering \textit{chicken coop}. In contrast, children often use association, e.g., answering \textit{egg}. This paper investigates whether large language models (LLMs) solve verbal analogies in A:B::C:? form using associations, similar to what children do. We use verbal analogies extracted from an online adaptive learning environment, where 14,002 7-12 year-olds from the Netherlands solved 622 analogies in Dutch. The six tested Dutch monolingual and multilingual LLMs performed around the same level as children, with MGPT performing worst, around the 7-year-old level, and XLM-V and GPT-3 the best, slightly above the 11-year-old level. However, when we control for associative processes this picture changes and each model's performance level drops 1-2 years. Further experiments demonstrate that associative processes often underlie correctly solved analogies. We conclude that the LLMs we tested indeed tend to solve verbal analogies by association with C like children do.

Related papers

• Can Large Language Models generalize analogy solving like people can? [46.02074643846298]
  In people, the ability to solve analogies such as "body : feet :: table :?" emerges in childhood. Recent research shows that large language models (LLMs) can solve various forms of analogies.
  arXiv  Detail & Related papers  (2024-11-04T18:18:38Z)
• Do Large Language Models Solve ARC Visual Analogies Like People Do? [0.0]
  We compared human and large language model (LLM) performance on a new child-friendly set of ARC items. Results show that both children and adults outperform most LLMs on these tasks.
  arXiv  Detail & Related papers  (2024-03-13T09:48:13Z)
• ParallelPARC: A Scalable Pipeline for Generating Natural-Language Analogies [16.92480305308536]
  We develop a pipeline for creating complex, paragraph-based analogies. We publish a gold-set, validated by humans, and a silver-set, generated automatically. We demonstrate that our silver-set is useful for training models.
  arXiv  Detail & Related papers  (2024-03-02T08:53:40Z)
• AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies [19.613777134600408]
  Analogical thinking allows humans to solve problems in creative ways. Can language models (LMs) do the same? benchmarking approach focuses on aspects of this ability that are common among humans.
  arXiv  Detail & Related papers  (2024-02-19T18:56:44Z)
• Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners? [140.9751389452011]
  We study the biases of large language models (LLMs) in relation to those known in children when solving arithmetic word problems. We generate a novel set of word problems for each of these tests, using a neuro-symbolic approach that enables fine-grained control over the problem features.
  arXiv  Detail & Related papers  (2024-01-31T18:48:20Z)
• StoryAnalogy: Deriving Story-level Analogies from Large Language Models to Unlock Analogical Understanding [72.38872974837462]
  We evaluate the ability to identify and generate analogies by constructing a first-of-its-kind large-scale story-level analogy corpus. textscStory Analogy contains 24K story pairs from diverse domains with human annotations on two similarities from the extended Structure-Mapping Theory. We observe that the data in textscStory Analogy can improve the quality of analogy generation in large language models.
  arXiv  Detail & Related papers  (2023-10-19T16:29:23Z)
• Can language models learn analogical reasoning? Investigating training objectives and comparisons to human performance [0.0]
  We test several ways to learn basic analogical reasoning, specifically focusing on analogies that are more typical of what is used to evaluate analogical reasoning in humans. Our experiments find that models are able to learn analogical reasoning, even with a small amount of data.
  arXiv  Detail & Related papers  (2023-10-09T10:34:38Z)
• ANALOGYKB: Unlocking Analogical Reasoning of Language Models with A Million-scale Knowledge Base [51.777618249271725]
  ANALOGYKB is a million-scale analogy knowledge base derived from existing knowledge graphs (KGs) It identifies two types of analogies from the KGs: 1) analogies of the same relations, which can be directly extracted from the KGs, and 2) analogies of analogous relations, which are identified with a selection and filtering pipeline enabled by large language models (LLMs)
  arXiv  Detail & Related papers  (2023-05-10T09:03:01Z)
• Speaking Multiple Languages Affects the Moral Bias of Language Models [70.94372902010232]
  Pre-trained multilingual language models (PMLMs) are commonly used when dealing with data from multiple languages and cross-lingual transfer. Do the models capture moral norms from English and impose them on other languages? Our experiments demonstrate that, indeed, PMLMs encode differing moral biases, but these do not necessarily correspond to cultural differences or commonalities in human opinions.
  arXiv  Detail & Related papers  (2022-11-14T20:08:54Z)
• Things not Written in Text: Exploring Spatial Commonsense from Visual Signals [77.46233234061758]
  We investigate whether models with visual signals learn more spatial commonsense than text-based models. We propose a benchmark that focuses on the relative scales of objects, and the positional relationship between people and objects under different actions. We find that image synthesis models are more capable of learning accurate and consistent spatial knowledge than other models.
  arXiv  Detail & Related papers  (2022-03-15T17:02:30Z)
• PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World [86.21137454228848]
  We factorize PIGLeT into a physical dynamics model, and a separate language model. PIGLeT can read a sentence, simulate neurally what might happen next, and then communicate that result through a literal symbolic representation. It is able to correctly forecast "what happens next" given an English sentence over 80% of the time, outperforming a 100x larger, text-to-text approach by over 10%.
  arXiv  Detail & Related papers  (2021-06-01T02:32:12Z)
• BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies? [35.381345454627]
  We analyze the capabilities of transformer-based language models on an unsupervised task of identifying analogies. Off-the-shelf language models can identify analogies to a certain extent, but struggle with abstract and complex relations. Our results raise important questions for future work about how, and to what extent, pre-trained language models capture knowledge about abstract semantic relations.
  arXiv  Detail & Related papers  (2021-05-11T11:38:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.