Contextualized Scene Imagination for Generative Commonsense Reasoning

• URL: http://arxiv.org/abs/2112.06318v1
• Date: Sun, 12 Dec 2021 20:38:08 GMT
• Title: Contextualized Scene Imagination for Generative Commonsense Reasoning
• Authors: PeiFeng Wang, Jonathan Zamora, Junfeng Liu, Filip Ilievski, Muhao Chen, Xiang Ren
• Abstract summary: generative commonsense reasoning skills are lacking in state-of-the-art text generation methods. We propose an Imagine-and-Verbalize (I&V) method, which learns to imagine a relational scene knowledge graph. Experiments demonstrate the effectiveness of I&V in improving language models on both concept-to-sentence and concept-to-story generation tasks.
• Score: 35.03682416576795
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Humans use natural language to compose common concepts from their environment into plausible, day-to-day scene descriptions. However, such generative commonsense reasoning (GCSR) skills are lacking in state-of-the-art text generation methods. Descriptive sentences about arbitrary concepts generated by neural text generation models (e.g., pre-trained text-to-text Transformers) are often grammatically fluent but may not correspond to human common sense, largely due to their lack of mechanisms to capture concept relations, to identify implicit concepts, and to perform generalizable reasoning about unseen concept compositions. In this paper, we propose an Imagine-and-Verbalize (I&V) method, which learns to imagine a relational scene knowledge graph (SKG) with relations between the input concepts, and leverage the SKG as a constraint when generating a plausible scene description. We collect and harmonize a set of knowledge resources from different domains and modalities, providing a rich auxiliary supervision signal for I&V. The experiments demonstrate the effectiveness of I&V in improving language models on both concept-to-sentence and concept-to-story generation tasks, while enabling the model to learn well from fewer task examples and generate SKGs that make common sense to human annotators.

Related papers

• Compositional Entailment Learning for Hyperbolic Vision-Language Models [54.41927525264365]
  We show how to fully leverage the innate hierarchical nature of hyperbolic embeddings by looking beyond individual image-text pairs. We propose Compositional Entailment Learning for hyperbolic vision-language models. Empirical evaluation on a hyperbolic vision-language model trained with millions of image-text pairs shows that the proposed compositional learning approach outperforms conventional Euclidean CLIP learning.
  arXiv  Detail & Related papers  (2024-10-09T14:12:50Z)
• CoLiDR: Concept Learning using Aggregated Disentangled Representations [29.932706137805713]
  Interpretability of Deep Neural Networks using concept-based models offers a promising way to explain model behavior through human-understandable concepts. A parallel line of research focuses on disentangling the data distribution into its underlying generative factors, in turn explaining the data generation process. While both directions have received extensive attention, little work has been done on explaining concepts in terms of generative factors to unify mathematically disentangled representations and human-understandable concepts.
  arXiv  Detail & Related papers  (2024-07-27T16:55:14Z)
• Neural Language of Thought Models [18.930227757853313]
  We introduce the Neural Language of Thought Model (NLoTM), a novel approach for unsupervised learning of LoTH-inspired representation and generation. NLoTM comprises two key components: (1) the Semantic Vector-Quantized Variational Autoencoder, which learns hierarchical, composable discrete representations aligned with objects and their properties, and (2) the Autoregressive LoT Prior, an autoregressive transformer that learns to generate semantic concept tokens compositionally. We evaluate NLoTM on several 2D and 3D image datasets, demonstrating superior performance in downstream tasks, out-of-distribution generalization, and image generation
  arXiv  Detail & Related papers  (2024-02-02T08:13:18Z)
• Interpreting Pretrained Language Models via Concept Bottlenecks [55.47515772358389]
  Pretrained language models (PLMs) have made significant strides in various natural language processing tasks. The lack of interpretability due to their black-box'' nature poses challenges for responsible implementation. We propose a novel approach to interpreting PLMs by employing high-level, meaningful concepts that are easily understandable for humans.
  arXiv  Detail & Related papers  (2023-11-08T20:41:18Z)
• A Probabilistic-Logic based Commonsense Representation Framework for Modelling Inferences with Multiple Antecedents and Varying Likelihoods [5.87677276882675]
  Commonsense knowledge-graphs (CKGs) are important resources towards building machines that can'reason' on text or environmental inputs and make inferences beyond perception. In this work, we study how commonsense knowledge can be better represented by -- (i) utilizing a probabilistic logic representation scheme to model composite inferential knowledge and represent conceptual beliefs with varying likelihoods, and (ii) incorporating a hierarchical conceptual ontology to identify salient concept-relevant relations and organize beliefs at different conceptual levels.
  arXiv  Detail & Related papers  (2022-11-30T08:44:30Z)
• ConceptX: A Framework for Latent Concept Analysis [21.760620298330235]
  We present ConceptX, a human-in-the-loop framework for interpreting and annotating latent representational space in Language Models (pLMs) We use an unsupervised method to discover concepts learned in these models and enable a graphical interface for humans to generate explanations for the concepts.
  arXiv  Detail & Related papers  (2022-11-12T11:31:09Z)
• Analyzing Encoded Concepts in Transformer Language Models [21.76062029833023]
  ConceptX analyses how latent concepts are encoded in representations learned within pre-trained language models. It uses clustering to discover the encoded concepts and explains them by aligning with a large set of human-defined concepts.
  arXiv  Detail & Related papers  (2022-06-27T13:32:10Z)
• Lexically-constrained Text Generation through Commonsense Knowledge Extraction and Injection [62.071938098215085]
  We focus on the Commongen benchmark, wherein the aim is to generate a plausible sentence for a given set of input concepts. We propose strategies for enhancing the semantic correctness of the generated text.
  arXiv  Detail & Related papers  (2020-12-19T23:23:40Z)
• Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs [106.15931418425906]
  We present the first study focused on generating natural language rationales across several complex visual reasoning tasks. We present RationaleVT Transformer, an integrated model that learns to generate free-text rationales by combining pretrained language models with object recognition, grounded visual semantic frames, and visual commonsense graphs. Our experiments show that the base pretrained language model benefits from visual adaptation and that free-text rationalization is a promising research direction to complement model interpretability for complex visual-textual reasoning tasks.
  arXiv  Detail & Related papers  (2020-10-15T05:08:56Z)
• KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning [78.81080813406177]
  We propose a novel knowledge graph augmented pre-trained language generation model KG-BART. KG-BART encompasses the complex relations of concepts through the knowledge graph and produces more logical and natural sentences as output.
  arXiv  Detail & Related papers  (2020-09-26T19:57:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.