Penguins Don't Fly: Reasoning about Generics through Instantiations and Exceptions

• URL: http://arxiv.org/abs/2205.11658v3
• Date: Fri, 24 Mar 2023 17:00:56 GMT
• Title: Penguins Don't Fly: Reasoning about Generics through Instantiations and Exceptions
• Authors: Emily Allaway, Jena D. Hwang, Chandra Bhagavatula, Kathleen McKeown, Doug Downey, Yejin Choi
• Abstract summary: We present a novel framework informed by linguistic theory to generate exemplars -- specific cases when a generic holds true or false. We generate 19k exemplars for 650 generics and show that our framework outperforms a strong GPT-3 baseline by 12.8 precision points.
• Score: 73.56753518339247
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generics express generalizations about the world (e.g., birds can fly) that are not universally true (e.g., newborn birds and penguins cannot fly). Commonsense knowledge bases, used extensively in NLP, encode some generic knowledge but rarely enumerate such exceptions and knowing when a generic statement holds or does not hold true is crucial for developing a comprehensive understanding of generics. We present a novel framework informed by linguistic theory to generate exemplars -- specific cases when a generic holds true or false. We generate ~19k exemplars for ~650 generics and show that our framework outperforms a strong GPT-3 baseline by 12.8 precision points. Our analysis highlights the importance of linguistic theory-based controllability for generating exemplars, the insufficiency of knowledge bases as a source of exemplars, and the challenges exemplars pose for the task of natural language inference.

Related papers

• GeniL: A Multilingual Dataset on Generalizing Language [19.43611224855484]
  Current methods to assess presence of stereotypes in generated language rely on simple template or co-occurrence based measures. We argue that understanding the sentential context is crucial for detecting instances of generalization. We build GeniL, a multilingual dataset of over 50K sentences from 9 languages annotated for instances of generalizations.
  arXiv  Detail & Related papers  (2024-04-08T20:58:06Z)
• SLOG: A Structural Generalization Benchmark for Semantic Parsing [68.19511282584304]
  The goal of compositional generalization benchmarks is to evaluate how well models generalize to new complex linguistic expressions. Existing benchmarks often focus on lexical generalization, the interpretation of novel lexical items in syntactic structures familiar from training, are often underrepresented. We introduce SLOG, a semantic parsing dataset that extends COGS with 17 structural generalization cases.
  arXiv  Detail & Related papers  (2023-10-23T15:39:09Z)
• Uncertainty in Natural Language Generation: From Theory to Applications [42.55924708592451]
  We argue that a principled treatment of uncertainty can assist in creating systems and evaluation protocols better aligned with these goals. We first present the fundamental theory, frameworks and vocabulary required to represent uncertainty. We then propose a two-dimensional taxonomy that is more informative and faithful than the popular aleatoric/epistemic dichotomy.
  arXiv  Detail & Related papers  (2023-07-28T17:51:21Z)
• A Measure-Theoretic Characterization of Tight Language Models [105.16477132329416]
  In some pathological cases, probability mass can leak'' onto the set of infinite sequences. This paper offers a measure-theoretic treatment of language modeling. We prove that many popular language model families are in fact tight, meaning that they will not leak in this sense.
  arXiv  Detail & Related papers  (2022-12-20T18:17:11Z)
• Formal Specifications from Natural Language [3.1806743741013657]
  We study the ability of language models to translate natural language into formal specifications with complex semantics. In particular, we fine-tune off-the-shelf language models on three datasets consisting of structured English sentences.
  arXiv  Detail & Related papers  (2022-06-04T10:49:30Z)
• Does BERT really agree ? Fine-grained Analysis of Lexical Dependence on a Syntactic Task [70.29624135819884]
  We study the extent to which BERT is able to perform lexically-independent subject-verb number agreement (NA) on targeted syntactic templates. Our results on nonce sentences suggest that the model generalizes well for simple templates, but fails to perform lexically-independent syntactic generalization when as little as one attractor is present.
  arXiv  Detail & Related papers  (2022-04-14T11:33:15Z)
• Leveraging the Inductive Bias of Large Language Models for Abstract Textual Reasoning [3.616948583169635]
  Large natural language models (such as GPT-3 or T5) demonstrate impressive abilities across a range of general NLP tasks. We show that the knowledge embedded in such models provides a useful inductive bias, not just on traditional NLP tasks, but also in the nontraditional task of training a symbolic reasoning engine.
  arXiv  Detail & Related papers  (2021-10-05T21:40:46Z)
• Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning [69.1137074774244]
  Leveraging language interactions effectively requires addressing limitations in the two most common approaches to language grounding. We introduce the idea of neural abstructions: a set of constraints on the inference procedure of a label-conditioned generative model. We show that with this method a user population is able to build a semantic modification for an open-ended house task in Minecraft.
  arXiv  Detail & Related papers  (2021-07-20T07:01:15Z)
• Infusing Finetuning with Semantic Dependencies [62.37697048781823]
  We show that, unlike syntax, semantics is not brought to the surface by today's pretrained models. We then use convolutional graph encoders to explicitly incorporate semantic parses into task-specific finetuning.
  arXiv  Detail & Related papers  (2020-12-10T01:27:24Z)
• GenericsKB: A Knowledge Base of Generic Statements [18.68800894936855]
  We present a new resource for the NLP community, namely a large (3.5M+ sentence) knowledge base of *generic statements* This is the first large resource to contain *naturally occurring* generic sentences, as opposed to extracted or crowdsourced triples. All GenericsKB sentences are annotated with their topical term, surrounding context (sentences), and a (learned) confidence.
  arXiv  Detail & Related papers  (2020-05-02T00:08:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.