Scientific and Creative Analogies in Pretrained Language Models

• URL: http://arxiv.org/abs/2211.15268v1
• Date: Mon, 28 Nov 2022 12:49:44 GMT
• Title: Scientific and Creative Analogies in Pretrained Language Models
• Authors: Tamara Czinczoll, Helen Yannakoudakis, Pushkar Mishra, Ekaterina Shutova
• Abstract summary: This paper examines the encoding of analogy in large-scale pretrained language models, such as BERT and GPT-2. We introduce the Scientific and Creative Analogy dataset (SCAN), a novel analogy dataset containing systematic mappings of multiple attributes and relational structures across dissimilar domains. We find that state-of-the-art LMs achieve low performance on these complex analogy tasks, highlighting the challenges still posed by analogy understanding.
• Score: 24.86477727507679
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper examines the encoding of analogy in large-scale pretrained language models, such as BERT and GPT-2. Existing analogy datasets typically focus on a limited set of analogical relations, with a high similarity of the two domains between which the analogy holds. As a more realistic setup, we introduce the Scientific and Creative Analogy dataset (SCAN), a novel analogy dataset containing systematic mappings of multiple attributes and relational structures across dissimilar domains. Using this dataset, we test the analogical reasoning capabilities of several widely-used pretrained language models (LMs). We find that state-of-the-art LMs achieve low performance on these complex analogy tasks, highlighting the challenges still posed by analogy understanding.

Related papers

• Explaining Text Similarity in Transformer Models [52.571158418102584]
  Recent advances in explainable AI have made it possible to mitigate limitations by leveraging improved explanations for Transformers. We use BiLRP, an extension developed for computing second-order explanations in bilinear similarity models, to investigate which feature interactions drive similarity in NLP models. Our findings contribute to a deeper understanding of different semantic similarity tasks and models, highlighting how novel explainable AI methods enable in-depth analyses and corpus-level insights.
  arXiv  Detail & Related papers  (2024-05-10T17:11:31Z)
• StoryAnalogy: Deriving Story-level Analogies from Large Language Models to Unlock Analogical Understanding [72.38872974837462]
  We evaluate the ability to identify and generate analogies by constructing a first-of-its-kind large-scale story-level analogy corpus. textscStory Analogy contains 24K story pairs from diverse domains with human annotations on two similarities from the extended Structure-Mapping Theory. We observe that the data in textscStory Analogy can improve the quality of analogy generation in large language models.
  arXiv  Detail & Related papers  (2023-10-19T16:29:23Z)
• ARN: Analogical Reasoning on Narratives [13.707344123755126]
  We develop a framework that operationalizes dominant theories of analogy, using narrative elements to create surface and system mappings. We show that while all LLMs can largely recognize near analogies, even the largest ones struggle with far analogies in a zero-shot setting.
  arXiv  Detail & Related papers  (2023-10-02T08:58:29Z)
• Why Do We Need Neuro-symbolic AI to Model Pragmatic Analogies? [6.8107181513711055]
  A hallmark of intelligence is the ability to use a familiar domain to make inferences about a less familiar domain, known as analogical reasoning. We discuss analogies at four distinct levels of complexity: lexical analogies, syntactic analogies, semantic analogies, and pragmatic analogies. We employ Neuro-symbolic AI techniques that combine statistical and symbolic AI, informing the representation of unstructured text to highlight and augment relevant content, provide abstraction and guide the mapping process.
  arXiv  Detail & Related papers  (2023-08-02T21:13:38Z)
• Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abduction [46.2032673640788]
  The vital role of analogical reasoning in human cognition allows us to grasp novel concepts by linking them with familiar ones through shared relational structures. This work suggests that Large Language Models (LLMs) often overlook the structures that underpin these analogies. This paper introduces a task of analogical structure abduction, grounded in cognitive psychology, designed to abduce structures that form an analogy between two systems.
  arXiv  Detail & Related papers  (2023-05-22T03:04:06Z)
• ANALOGYKB: Unlocking Analogical Reasoning of Language Models with A Million-scale Knowledge Base [51.777618249271725]
  ANALOGYKB is a million-scale analogy knowledge base derived from existing knowledge graphs (KGs) It identifies two types of analogies from the KGs: 1) analogies of the same relations, which can be directly extracted from the KGs, and 2) analogies of analogous relations, which are identified with a selection and filtering pipeline enabled by large language models (LLMs)
  arXiv  Detail & Related papers  (2023-05-10T09:03:01Z)
• BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies? [35.381345454627]
  We analyze the capabilities of transformer-based language models on an unsupervised task of identifying analogies. Off-the-shelf language models can identify analogies to a certain extent, but struggle with abstract and complex relations. Our results raise important questions for future work about how, and to what extent, pre-trained language models capture knowledge about abstract semantic relations.
  arXiv  Detail & Related papers  (2021-05-11T11:38:49Z)
• Few-Shot Named Entity Recognition: A Comprehensive Study [92.40991050806544]
  We investigate three schemes to improve the model generalization ability for few-shot settings. We perform empirical comparisons on 10 public NER datasets with various proportions of labeled data. We create new state-of-the-art results on both few-shot and training-free settings.
  arXiv  Detail & Related papers  (2020-12-29T23:43:16Z)
• Few-shot Visual Reasoning with Meta-analogical Contrastive Learning [141.2562447971]
  We propose to solve a few-shot (or low-shot) visual reasoning problem, by resorting to analogical reasoning. We extract structural relationships between elements in both domains, and enforce them to be as similar as possible with analogical learning. We validate our method on RAVEN dataset, on which it outperforms state-of-the-art method, with larger gains when the training data is scarce.
  arXiv  Detail & Related papers  (2020-07-23T14:00:34Z)
• Learning to See Analogies: A Connectionist Exploration [0.0]
  This dissertation explores the integration of learning and analogy-making through the development of a computer program, called Analogator. By "seeing" many different analogy problems, along with possible solutions, Analogator gradually develops an ability to make new analogies.
  arXiv  Detail & Related papers  (2020-01-18T14:06:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.