ANALOGYKB: Unlocking Analogical Reasoning of Language Models with A Million-scale Knowledge Base

• URL: http://arxiv.org/abs/2305.05994v2
• Date: Fri, 17 May 2024 07:59:19 GMT
• Title: ANALOGYKB: Unlocking Analogical Reasoning of Language Models with A Million-scale Knowledge Base
• Authors: Siyu Yuan, Jiangjie Chen, Changzhi Sun, Jiaqing Liang, Yanghua Xiao, Deqing Yang,
• Abstract summary: ANALOGYKB is a million-scale analogy knowledge base derived from existing knowledge graphs (KGs) It identifies two types of analogies from the KGs: 1) analogies of the same relations, which can be directly extracted from the KGs, and 2) analogies of analogous relations, which are identified with a selection and filtering pipeline enabled by large language models (LLMs)
• Score: 51.777618249271725
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Analogical reasoning is a fundamental cognitive ability of humans. However, current language models (LMs) still struggle to achieve human-like performance in analogical reasoning tasks due to a lack of resources for model training. In this work, we address this gap by proposing ANALOGYKB, a million-scale analogy knowledge base (KB) derived from existing knowledge graphs (KGs). ANALOGYKB identifies two types of analogies from the KGs: 1) analogies of the same relations, which can be directly extracted from the KGs, and 2) analogies of analogous relations, which are identified with a selection and filtering pipeline enabled by large language models (LLMs), followed by minor human efforts for data quality control. Evaluations on a series of datasets of two analogical reasoning tasks (analogy recognition and generation) demonstrate that ANALOGYKB successfully enables both smaller LMs and LLMs to gain better analogical reasoning capabilities.

Related papers

• Explaining Text Similarity in Transformer Models [52.571158418102584]
  Recent advances in explainable AI have made it possible to mitigate limitations by leveraging improved explanations for Transformers. We use BiLRP, an extension developed for computing second-order explanations in bilinear similarity models, to investigate which feature interactions drive similarity in NLP models. Our findings contribute to a deeper understanding of different semantic similarity tasks and models, highlighting how novel explainable AI methods enable in-depth analyses and corpus-level insights.
  arXiv  Detail & Related papers  (2024-05-10T17:11:31Z)
• CausalGym: Benchmarking causal interpretability methods on linguistic tasks [52.61917615039112]
  We use CausalGym to benchmark the ability of interpretability methods to causally affect model behaviour. We study the pythia models (14M--6.9B) and assess the causal efficacy of a wide range of interpretability methods. We find that DAS outperforms the other methods, and so we use it to study the learning trajectory of two difficult linguistic phenomena.
  arXiv  Detail & Related papers  (2024-02-19T21:35:56Z)
• StoryAnalogy: Deriving Story-level Analogies from Large Language Models to Unlock Analogical Understanding [72.38872974837462]
  We evaluate the ability to identify and generate analogies by constructing a first-of-its-kind large-scale story-level analogy corpus. textscStory Analogy contains 24K story pairs from diverse domains with human annotations on two similarities from the extended Structure-Mapping Theory. We observe that the data in textscStory Analogy can improve the quality of analogy generation in large language models.
  arXiv  Detail & Related papers  (2023-10-19T16:29:23Z)
• Can language models learn analogical reasoning? Investigating training objectives and comparisons to human performance [0.0]
  We test several ways to learn basic analogical reasoning, specifically focusing on analogies that are more typical of what is used to evaluate analogical reasoning in humans. Our experiments find that models are able to learn analogical reasoning, even with a small amount of data.
  arXiv  Detail & Related papers  (2023-10-09T10:34:38Z)
• ARN: Analogical Reasoning on Narratives [13.707344123755126]
  We develop a framework that operationalizes dominant theories of analogy, using narrative elements to create surface and system mappings. We show that while all LLMs can largely recognize near analogies, even the largest ones struggle with far analogies in a zero-shot setting.
  arXiv  Detail & Related papers  (2023-10-02T08:58:29Z)
• Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abduction [46.2032673640788]
  The vital role of analogical reasoning in human cognition allows us to grasp novel concepts by linking them with familiar ones through shared relational structures. This work suggests that Large Language Models (LLMs) often overlook the structures that underpin these analogies. This paper introduces a task of analogical structure abduction, grounded in cognitive psychology, designed to abduce structures that form an analogy between two systems.
  arXiv  Detail & Related papers  (2023-05-22T03:04:06Z)
• Scientific and Creative Analogies in Pretrained Language Models [24.86477727507679]
  This paper examines the encoding of analogy in large-scale pretrained language models, such as BERT and GPT-2. We introduce the Scientific and Creative Analogy dataset (SCAN), a novel analogy dataset containing systematic mappings of multiple attributes and relational structures across dissimilar domains. We find that state-of-the-art LMs achieve low performance on these complex analogy tasks, highlighting the challenges still posed by analogy understanding.
  arXiv  Detail & Related papers  (2022-11-28T12:49:44Z)
• Analogy Generation by Prompting Large Language Models: A Case Study of InstructGPT [47.81902362511172]
  We propose a novel application of prompting Pre-trained Language Models (PLMs) to generate analogies. We found that it is feasible to prompt InstructGPT to generate meaningful analogies and the best prompts tend to be precise imperative statements. We also conducted human evaluation on 1.4k of the generated analogies and found that the quality of generations varies substantially by model size.
  arXiv  Detail & Related papers  (2022-10-09T06:35:14Z)
• Tackling Morphological Analogies Using Deep Learning -- Extended Version [8.288496996031684]
  Analogical proportions are statements of the form "A is to B as C is to D" We propose an approach using deep learning to detect and solve morphological analogies. We demonstrate our model's competitive performance on analogy detection and resolution over multiple languages.
  arXiv  Detail & Related papers  (2021-11-09T13:45:23Z)
• BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies? [35.381345454627]
  We analyze the capabilities of transformer-based language models on an unsupervised task of identifying analogies. Off-the-shelf language models can identify analogies to a certain extent, but struggle with abstract and complex relations. Our results raise important questions for future work about how, and to what extent, pre-trained language models capture knowledge about abstract semantic relations.
  arXiv  Detail & Related papers  (2021-05-11T11:38:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.