Contextualized Scene Imagination for Generative Commonsense Reasoning
- URL: http://arxiv.org/abs/2112.06318v1
- Date: Sun, 12 Dec 2021 20:38:08 GMT
- Title: Contextualized Scene Imagination for Generative Commonsense Reasoning
- Authors: PeiFeng Wang, Jonathan Zamora, Junfeng Liu, Filip Ilievski, Muhao
Chen, Xiang Ren
- Abstract summary: generative commonsense reasoning skills are lacking in state-of-the-art text generation methods.
We propose an Imagine-and-Verbalize (I&V) method, which learns to imagine a relational scene knowledge graph.
Experiments demonstrate the effectiveness of I&V in improving language models on both concept-to-sentence and concept-to-story generation tasks.
- Score: 35.03682416576795
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Humans use natural language to compose common concepts from their environment
into plausible, day-to-day scene descriptions. However, such generative
commonsense reasoning (GCSR) skills are lacking in state-of-the-art text
generation methods. Descriptive sentences about arbitrary concepts generated by
neural text generation models (e.g., pre-trained text-to-text Transformers) are
often grammatically fluent but may not correspond to human common sense,
largely due to their lack of mechanisms to capture concept relations, to
identify implicit concepts, and to perform generalizable reasoning about unseen
concept compositions. In this paper, we propose an Imagine-and-Verbalize (I&V)
method, which learns to imagine a relational scene knowledge graph (SKG) with
relations between the input concepts, and leverage the SKG as a constraint when
generating a plausible scene description. We collect and harmonize a set of
knowledge resources from different domains and modalities, providing a rich
auxiliary supervision signal for I&V. The experiments demonstrate the
effectiveness of I&V in improving language models on both concept-to-sentence
and concept-to-story generation tasks, while enabling the model to learn well
from fewer task examples and generate SKGs that make common sense to human
annotators.
Related papers
- Learning Structure and Knowledge Aware Representation with Large Language Models for Concept Recommendation [50.31872005772817]
Concept recommendation aims to suggest the next concept for learners to study based on their knowledge states and the human knowledge system.
Previous approaches have not effectively integrated the human knowledge system into the process of designing these educational models.
We propose a novel Structure and Knowledge Aware Representation learning framework for concept Recommendation (SKarREC)
arXiv Detail & Related papers (2024-05-21T01:35:36Z) - Neural Language of Thought Models [18.930227757853313]
We introduce the Neural Language of Thought Model (NLoTM), a novel approach for unsupervised learning of LoTH-inspired representation and generation.
NLoTM comprises two key components: (1) the Semantic Vector-Quantized Variational Autoencoder, which learns hierarchical, composable discrete representations aligned with objects and their properties, and (2) the Autoregressive LoT Prior, an autoregressive transformer that learns to generate semantic concept tokens compositionally.
We evaluate NLoTM on several 2D and 3D image datasets, demonstrating superior performance in downstream tasks, out-of-distribution generalization, and image generation
arXiv Detail & Related papers (2024-02-02T08:13:18Z) - Language-Informed Visual Concept Learning [22.911347501969857]
We train a set of concept encoders to encode the information pertinent to a set of language-informed concept axes.
We then anchor the concept embeddings to a set of text embeddings obtained from a pre-trained Visual Question Answering (VQA) model.
At inference time, the model extracts concept embeddings along various axes from new test images, which can be remixed to generate images with novel compositions of visual concepts.
arXiv Detail & Related papers (2023-12-06T16:24:47Z) - Interpreting Pretrained Language Models via Concept Bottlenecks [55.47515772358389]
Pretrained language models (PLMs) have made significant strides in various natural language processing tasks.
The lack of interpretability due to their black-box'' nature poses challenges for responsible implementation.
We propose a novel approach to interpreting PLMs by employing high-level, meaningful concepts that are easily understandable for humans.
arXiv Detail & Related papers (2023-11-08T20:41:18Z) - A Probabilistic-Logic based Commonsense Representation Framework for
Modelling Inferences with Multiple Antecedents and Varying Likelihoods [5.87677276882675]
Commonsense knowledge-graphs (CKGs) are important resources towards building machines that can'reason' on text or environmental inputs and make inferences beyond perception.
In this work, we study how commonsense knowledge can be better represented by -- (i) utilizing a probabilistic logic representation scheme to model composite inferential knowledge and represent conceptual beliefs with varying likelihoods, and (ii) incorporating a hierarchical conceptual ontology to identify salient concept-relevant relations and organize beliefs at different conceptual levels.
arXiv Detail & Related papers (2022-11-30T08:44:30Z) - ConceptX: A Framework for Latent Concept Analysis [21.760620298330235]
We present ConceptX, a human-in-the-loop framework for interpreting and annotating latent representational space in Language Models (pLMs)
We use an unsupervised method to discover concepts learned in these models and enable a graphical interface for humans to generate explanations for the concepts.
arXiv Detail & Related papers (2022-11-12T11:31:09Z) - Analyzing Encoded Concepts in Transformer Language Models [21.76062029833023]
ConceptX analyses how latent concepts are encoded in representations learned within pre-trained language models.
It uses clustering to discover the encoded concepts and explains them by aligning with a large set of human-defined concepts.
arXiv Detail & Related papers (2022-06-27T13:32:10Z) - Lexically-constrained Text Generation through Commonsense Knowledge
Extraction and Injection [62.071938098215085]
We focus on the Commongen benchmark, wherein the aim is to generate a plausible sentence for a given set of input concepts.
We propose strategies for enhancing the semantic correctness of the generated text.
arXiv Detail & Related papers (2020-12-19T23:23:40Z) - Natural Language Rationales with Full-Stack Visual Reasoning: From
Pixels to Semantic Frames to Commonsense Graphs [106.15931418425906]
We present the first study focused on generating natural language rationales across several complex visual reasoning tasks.
We present RationaleVT Transformer, an integrated model that learns to generate free-text rationales by combining pretrained language models with object recognition, grounded visual semantic frames, and visual commonsense graphs.
Our experiments show that the base pretrained language model benefits from visual adaptation and that free-text rationalization is a promising research direction to complement model interpretability for complex visual-textual reasoning tasks.
arXiv Detail & Related papers (2020-10-15T05:08:56Z) - KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense
Reasoning [78.81080813406177]
We propose a novel knowledge graph augmented pre-trained language generation model KG-BART.
KG-BART encompasses the complex relations of concepts through the knowledge graph and produces more logical and natural sentences as output.
arXiv Detail & Related papers (2020-09-26T19:57:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.