ParallelPARC: A Scalable Pipeline for Generating Natural-Language Analogies

• URL: http://arxiv.org/abs/2403.01139v4
• Date: Tue, 14 May 2024 16:41:24 GMT
• Title: ParallelPARC: A Scalable Pipeline for Generating Natural-Language Analogies
• Authors: Oren Sultan, Yonatan Bitton, Ron Yosef, Dafna Shahaf,
• Abstract summary: We develop a pipeline for creating complex, paragraph-based analogies. We publish a gold-set, validated by humans, and a silver-set, generated automatically. We demonstrate that our silver-set is useful for training models.
• Score: 16.92480305308536
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Analogy-making is central to human cognition, allowing us to adapt to novel situations -- an ability that current AI systems still lack. Most analogy datasets today focus on simple analogies (e.g., word analogies); datasets including complex types of analogies are typically manually curated and very small. We believe that this holds back progress in computational analogy. In this work, we design a data generation pipeline, ParallelPARC (Parallel Paragraph Creator) leveraging state-of-the-art Large Language Models (LLMs) to create complex, paragraph-based analogies, as well as distractors, both simple and challenging. We demonstrate our pipeline and create ProPara-Logy, a dataset of analogies between scientific processes. We publish a gold-set, validated by humans, and a silver-set, generated automatically. We test LLMs' and humans' analogy recognition in binary and multiple-choice settings, and found that humans outperform the best models (~13% gap) after a light supervision. We demonstrate that our silver-set is useful for training models. Lastly, we show challenging distractors confuse LLMs, but not humans. We hope our pipeline will encourage research in this emerging field.

Related papers

• Autonomous Vehicle Controllers From End-to-End Differentiable Simulation [60.05963742334746]
  We propose a differentiable simulator and design an analytic policy gradients (APG) approach to training AV controllers. Our proposed framework brings the differentiable simulator into an end-to-end training loop, where gradients of environment dynamics serve as a useful prior to help the agent learn a more grounded policy. We find significant improvements in performance and robustness to noise in the dynamics, as well as overall more intuitive human-like handling.
  arXiv  Detail & Related papers  (2024-09-12T11:50:06Z)
• Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines [74.42485647685272]
  We focus on Generative Masked Language Models (GMLMs) We train a model to fit conditional probabilities of the data distribution via masking, which are subsequently used as inputs to a Markov Chain to draw samples from the model. We adapt the T5 model for iteratively-refined parallel decoding, achieving 2-3x speedup in machine translation with minimal sacrifice in quality.
  arXiv  Detail & Related papers  (2024-07-22T18:00:00Z)
• AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies [19.613777134600408]
  Analogical thinking allows humans to solve problems in creative ways. Can language models (LMs) do the same? benchmarking approach focuses on aspects of this ability that are common among humans.
  arXiv  Detail & Related papers  (2024-02-19T18:56:44Z)
• StoryAnalogy: Deriving Story-level Analogies from Large Language Models to Unlock Analogical Understanding [72.38872974837462]
  We evaluate the ability to identify and generate analogies by constructing a first-of-its-kind large-scale story-level analogy corpus. textscStory Analogy contains 24K story pairs from diverse domains with human annotations on two similarities from the extended Structure-Mapping Theory. We observe that the data in textscStory Analogy can improve the quality of analogy generation in large language models.
  arXiv  Detail & Related papers  (2023-10-19T16:29:23Z)
• ARN: Analogical Reasoning on Narratives [13.707344123755126]
  We develop a framework that operationalizes dominant theories of analogy, using narrative elements to create surface and system mappings. We show that while all LLMs can largely recognize near analogies, even the largest ones struggle with far analogies in a zero-shot setting.
  arXiv  Detail & Related papers  (2023-10-02T08:58:29Z)
• Why Do We Need Neuro-symbolic AI to Model Pragmatic Analogies? [6.8107181513711055]
  A hallmark of intelligence is the ability to use a familiar domain to make inferences about a less familiar domain, known as analogical reasoning. We discuss analogies at four distinct levels of complexity: lexical analogies, syntactic analogies, semantic analogies, and pragmatic analogies. We employ Neuro-symbolic AI techniques that combine statistical and symbolic AI, informing the representation of unstructured text to highlight and augment relevant content, provide abstraction and guide the mapping process.
  arXiv  Detail & Related papers  (2023-08-02T21:13:38Z)
• ANALOGYKB: Unlocking Analogical Reasoning of Language Models with A Million-scale Knowledge Base [51.777618249271725]
  ANALOGYKB is a million-scale analogy knowledge base derived from existing knowledge graphs (KGs) It identifies two types of analogies from the KGs: 1) analogies of the same relations, which can be directly extracted from the KGs, and 2) analogies of analogous relations, which are identified with a selection and filtering pipeline enabled by large language models (LLMs)
  arXiv  Detail & Related papers  (2023-05-10T09:03:01Z)
• Scientific and Creative Analogies in Pretrained Language Models [24.86477727507679]
  This paper examines the encoding of analogy in large-scale pretrained language models, such as BERT and GPT-2. We introduce the Scientific and Creative Analogy dataset (SCAN), a novel analogy dataset containing systematic mappings of multiple attributes and relational structures across dissimilar domains. We find that state-of-the-art LMs achieve low performance on these complex analogy tasks, highlighting the challenges still posed by analogy understanding.
  arXiv  Detail & Related papers  (2022-11-28T12:49:44Z)
• Simulation-Based Parallel Training [55.41644538483948]
  We present our ongoing work to design a training framework that alleviates those bottlenecks. It generates data in parallel with the training process. We present a strategy to mitigate this bias with a memory buffer.
  arXiv  Detail & Related papers  (2022-11-08T09:31:25Z)
• Life is a Circus and We are the Clowns: Automatically Finding Analogies between Situations and Processes [12.8252101640812]
  Much research has suggested that analogies are key to non-brittle systems that can adapt to new domains. Despite their importance, analogies received little attention in the NLP community.
  arXiv  Detail & Related papers  (2022-10-21T18:54:17Z)
• Unsupervised Paraphrasing with Pretrained Language Models [85.03373221588707]
  We propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting. Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking. We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair and the ParaNMT datasets.
  arXiv  Detail & Related papers  (2020-10-24T11:55:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.