Lexically-constrained Text Generation through Commonsense Knowledge Extraction and Injection

• URL: http://arxiv.org/abs/2012.10813v1
• Date: Sat, 19 Dec 2020 23:23:40 GMT
• Title: Lexically-constrained Text Generation through Commonsense Knowledge Extraction and Injection
• Authors: Yikang Li, Pulkit Goel, Varsha Kuppur Rajendra, Har Simrat Singh, Jonathan Francis, Kaixin Ma, Eric Nyberg, Alessandro Oltramari
• Abstract summary: We focus on the Commongen benchmark, wherein the aim is to generate a plausible sentence for a given set of input concepts. We propose strategies for enhancing the semantic correctness of the generated text.
• Score: 62.071938098215085
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conditional text generation has been a challenging task that is yet to see human-level performance from state-of-the-art models. In this work, we specifically focus on the Commongen benchmark, wherein the aim is to generate a plausible sentence for a given set of input concepts. Despite advances in other tasks, large pre-trained language models that are fine-tuned on this dataset often produce sentences that are syntactically correct but qualitatively deviate from a human understanding of common sense. Furthermore, generated sequences are unable to fulfill such lexical requirements as matching part-of-speech and full concept coverage. In this paper, we explore how commonsense knowledge graphs can enhance model performance, with respect to commonsense reasoning and lexically-constrained decoding. We propose strategies for enhancing the semantic correctness of the generated text, which we accomplish through: extracting commonsense relations from Conceptnet, injecting these relations into the Unified Language Model (UniLM) through attention mechanisms, and enforcing the aforementioned lexical requirements through output constraints. By performing several ablations, we find that commonsense injection enables the generation of sentences that are more aligned with human understanding, while remaining compliant with lexical requirements.

Related papers

• XPrompt:Explaining Large Language Model's Generation via Joint Prompt Attribution [26.639271355209104]
  Large Language Models (LLMs) have demonstrated impressive performances in complex text generation tasks. The contribution of the input prompt to the generated content still remains obscure to humans. We introduce a counterfactual explanation framework based on joint prompt attribution, XPrompt.
  arXiv  Detail & Related papers  (2024-05-30T18:16:41Z)
• How Well Do Text Embedding Models Understand Syntax? [50.440590035493074]
  The ability of text embedding models to generalize across a wide range of syntactic contexts remains under-explored. Our findings reveal that existing text embedding models have not sufficiently addressed these syntactic understanding challenges. We propose strategies to augment the generalization ability of text embedding models in diverse syntactic scenarios.
  arXiv  Detail & Related papers  (2023-11-14T08:51:00Z)
• SituatedGen: Incorporating Geographical and Temporal Contexts into Generative Commonsense Reasoning [55.44065342292846]
  Generative commonsense reasoning is the task that requires machines, given a group of keywords, to compose a single coherent sentence with commonsense plausibility. We formalize this challenging task as SituatedGen, where machines with commonsense should generate a pair of contrastive sentences given a group of keywords including geographical or temporal entities. Experiments show that state-of-the-art generative language models struggle to generate sentences with commonsense plausibility and still lag far behind human performance.
  arXiv  Detail & Related papers  (2023-06-21T20:36:55Z)
• Discontinuous Grammar as a Foreign Language [0.7412445894287709]
  We extend the framework of sequence-to-sequence models for constituent parsing. We design several novelizations that can fully produce discontinuities. For the first time, we test a sequence-to-sequence model on the main discontinuous benchmarks.
  arXiv  Detail & Related papers  (2021-10-20T08:58:02Z)
• Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence [59.51720326054546]
  We propose a long text generation model, which can represent the prefix sentences at sentence level and discourse level in the decoding process. Our model can generate more coherent texts than state-of-the-art baselines.
  arXiv  Detail & Related papers  (2021-05-19T07:29:08Z)
• Nutribullets Hybrid: Multi-document Health Summarization [36.95954983680022]
  We present a method for generating comparative summaries that highlights similarities and contradictions in input documents. Our framework leads to more faithful, relevant and aggregation-sensitive summarization -- while being equally fluent.
  arXiv  Detail & Related papers  (2021-04-08T01:44:29Z)
• Infusing Finetuning with Semantic Dependencies [62.37697048781823]
  We show that, unlike syntax, semantics is not brought to the surface by today's pretrained models. We then use convolutional graph encoders to explicitly incorporate semantic parses into task-specific finetuning.
  arXiv  Detail & Related papers  (2020-12-10T01:27:24Z)
• Commonsense Evidence Generation and Injection in Reading Comprehension [57.31927095547153]
  We propose a Commonsense Evidence Generation and Injection framework in reading comprehension, named CEGI. The framework injects two kinds of auxiliary commonsense evidence into comprehensive reading to equip the machine with the ability of rational thinking. Experiments on the CosmosQA dataset demonstrate that the proposed CEGI model outperforms the current state-of-the-art approaches.
  arXiv  Detail & Related papers  (2020-05-11T16:31:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.