ePiC: Employing Proverbs in Context as a Benchmark for Abstract Language Understanding

• URL: http://arxiv.org/abs/2109.06838v2
• Date: Wed, 15 Sep 2021 15:50:33 GMT
• Title: ePiC: Employing Proverbs in Context as a Benchmark for Abstract Language Understanding
• Authors: Sayan Ghosh and Shashank Srivastava
• Abstract summary: We introduce a high-quality crowdsourced dataset of narratives for employing proverbs in context as a benchmark for abstract language understanding. The dataset provides fine-grained annotation of aligned spans between proverbs and narratives, and contains minimal lexical overlaps between narratives and proverbs. Our experiments show that neural language models struggle in our tasks compared to humans, and the tasks pose multiple learning challenges.
• Score: 12.107259467873092
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While large language models have shown exciting progress on several NLP benchmarks, evaluating their ability for complex analogical reasoning remains under-explored. Here, we introduce a high-quality crowdsourced dataset of narratives for employing proverbs in context as a benchmark for abstract language understanding. The dataset provides fine-grained annotation of aligned spans between proverbs and narratives, and contains minimal lexical overlaps between narratives and proverbs, ensuring that models need to go beyond surface-level reasoning to succeed. We explore three tasks: (1) proverb recommendation and alignment prediction, (2) narrative generation for a given proverb and topic, and (3) identifying narratives with similar motifs. Our experiments show that neural language models struggle in our tasks compared to humans, and the tasks pose multiple learning challenges.

Related papers

• Generating Continuations in Multilingual Idiomatic Contexts [2.0849578298972835]
  We test the ability of generative language models (LMs) in understanding nuanced language containing non-compositional figurative text. We conduct experiments using datasets in two distinct languages (English and Portuguese) under three different training settings. Our results suggest that the models are only slightly better at generating continuations for literal contexts than idiomatic contexts, with exceedingly small margins.
  arXiv  Detail & Related papers  (2023-10-31T05:40:33Z)
• Multi-level Contrastive Learning for Script-based Character Understanding [14.341307979533871]
  We tackle the scenario of understanding characters in scripts, which aims to learn the characters' personalities and identities from their utterances. We propose a multi-level contrastive learning framework to capture characters' global information in a fine-grained manner.
  arXiv  Detail & Related papers  (2023-10-20T02:40:52Z)
• DiPlomat: A Dialogue Dataset for Situated Pragmatic Reasoning [89.92601337474954]
  Pragmatic reasoning plays a pivotal role in deciphering implicit meanings that frequently arise in real-life conversations. We introduce a novel challenge, DiPlomat, aiming at benchmarking machines' capabilities on pragmatic reasoning and situated conversational understanding.
  arXiv  Detail & Related papers  (2023-06-15T10:41:23Z)
• Effective Cross-Task Transfer Learning for Explainable Natural Language Inference with T5 [50.574918785575655]
  We compare sequential fine-tuning with a model for multi-task learning in the context of boosting performance on two tasks. Our results show that while sequential multi-task learning can be tuned to be good at the first of two target tasks, it performs less well on the second and additionally struggles with overfitting.
  arXiv  Detail & Related papers  (2022-10-31T13:26:08Z)
• Don't Copy the Teacher: Data and Model Challenges in Embodied Dialogue [92.01165203498299]
  Embodied dialogue instruction following requires an agent to complete a complex sequence of tasks from a natural language exchange. This paper argues that imitation learning (IL) and related low-level metrics are actually misleading and do not align with the goals of embodied dialogue research.
  arXiv  Detail & Related papers  (2022-10-10T05:51:40Z)
• Coarse-to-Fine: Hierarchical Multi-task Learning for Natural Language Understanding [51.31622274823167]
  We propose a hierarchical framework with a coarse-to-fine paradigm, with the bottom level shared to all the tasks, the mid-level divided to different groups, and the top-level assigned to each of the tasks. This allows our model to learn basic language properties from all tasks, boost performance on relevant tasks, and reduce the negative impact from irrelevant tasks.
  arXiv  Detail & Related papers  (2022-08-19T02:46:20Z)
• Local Interpretations for Explainable Natural Language Processing: A Survey [5.717407321642629]
  This work investigates various methods to improve the interpretability of deep neural networks for Natural Language Processing (NLP) tasks. We provide a comprehensive discussion on the definition of the term interpretability and its various aspects at the beginning of this work.
  arXiv  Detail & Related papers  (2021-03-20T02:28:33Z)
• Multi-turn Dialogue Reading Comprehension with Pivot Turns and Knowledge [43.352833140317486]
  Multi-turn dialogue reading comprehension aims to teach machines to read dialogue contexts and solve tasks such as response selection and answering questions. This work makes the first attempt to tackle the above two challenges by extracting substantially important turns as pivot utterances. We propose a pivot-oriented deep selection model (PoDS) on top of the Transformer-based language models for dialogue comprehension.
  arXiv  Detail & Related papers  (2021-02-10T15:00:12Z)
• Probing Task-Oriented Dialogue Representation from Language Models [106.02947285212132]
  This paper investigates pre-trained language models to find out which model intrinsically carries the most informative representation for task-oriented dialogue tasks. We fine-tune a feed-forward layer as the classifier probe on top of a fixed pre-trained language model with annotated labels in a supervised way.
  arXiv  Detail & Related papers  (2020-10-26T21:34:39Z)
• Exemplar-Controllable Paraphrasing and Translation using Bitext [57.92051459102902]
  We adapt models from prior work to be able to learn solely from bilingual text (bitext) Our single proposed model can perform four tasks: controlled paraphrase generation in both languages and controlled machine translation in both language directions.
  arXiv  Detail & Related papers  (2020-10-12T17:02:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.