CIS2: A Simplified Commonsense Inference Evaluation for Story Prose

• URL: http://arxiv.org/abs/2202.07880v1
• Date: Wed, 16 Feb 2022 06:14:37 GMT
• Title: CIS2: A Simplified Commonsense Inference Evaluation for Story Prose
• Authors: Bryan Li, Lara J. Martin, and Chris Callison-Burch
• Abstract summary: We look at the domain of commonsense reasoning within story prose, which we call contextual commonsense inference (CCI) We introduce the task contextual commonsense inference in sentence selection (CIS$2$), a simplified task that avoids conflation by eliminating language generation altogether.
• Score: 21.32351425259654
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformers have been showing near-human performance on a variety of tasks, but they are not without their limitations. We discuss the issue of conflating results of transformers that are instructed to do multiple tasks simultaneously. In particular, we focus on the domain of commonsense reasoning within story prose, which we call contextual commonsense inference (CCI). We look at the GLUCOSE (Mostafazadeh et al 2020) dataset and task for predicting implicit commonsense inferences between story sentences. Since the GLUCOSE task simultaneously generates sentences and predicts the CCI relation, there is a conflation in the results. Is the model really measuring CCI or is its ability to generate grammatical text carrying the results? In this paper, we introduce the task contextual commonsense inference in sentence selection (CIS$^2$), a simplified task that avoids conflation by eliminating language generation altogether. Our findings emphasize the necessity of future work to disentangle language generation from the desired NLP tasks at hand.

Related papers

• Can Language Models Take A Hint? Prompting for Controllable Contextualized Commonsense Inference [12.941933077524919]
  We introduce "hinting," a data augmentation technique that enhances contextualized commonsense inference. "Hinting" employs a prefix prompting strategy using both hard and soft prompts to guide the inference process. Our results show that "hinting" does not compromise the performance of contextual commonsense inference while offering improved controllability.
  arXiv  Detail & Related papers  (2024-10-03T04:32:46Z)
• Complex Reasoning over Logical Queries on Commonsense Knowledge Graphs [61.796960984541464]
  We present COM2 (COMplex COMmonsense), a new dataset created by sampling logical queries. We verbalize them using handcrafted rules and large language models into multiple-choice and text generation questions. Experiments show that language models trained on COM2 exhibit significant improvements in complex reasoning ability.
  arXiv  Detail & Related papers  (2024-03-12T08:13:52Z)
• Towards Event Extraction from Speech with Contextual Clues [61.164413398231254]
  We introduce the Speech Event Extraction (SpeechEE) task and construct three synthetic training sets and one human-spoken test set. Compared to event extraction from text, SpeechEE poses greater challenges mainly due to complex speech signals that are continuous and have no word boundaries. Our method brings significant improvements on all datasets, achieving a maximum F1 gain of 10.7%.
  arXiv  Detail & Related papers  (2024-01-27T11:07:19Z)
• Adversarial Transformer Language Models for Contextual Commonsense Inference [14.12019824666882]
  Contextualized or discourse aware commonsense inference is the task of generating coherent commonsense assertions. Some problems with the task are: lack of controllability for topics of the inferred facts; lack of commonsense knowledge during training. We develop techniques to address the aforementioned problems in the task.
  arXiv  Detail & Related papers  (2023-02-10T18:21:13Z)
• Effective Cross-Task Transfer Learning for Explainable Natural Language Inference with T5 [50.574918785575655]
  We compare sequential fine-tuning with a model for multi-task learning in the context of boosting performance on two tasks. Our results show that while sequential multi-task learning can be tuned to be good at the first of two target tasks, it performs less well on the second and additionally struggles with overfitting.
  arXiv  Detail & Related papers  (2022-10-31T13:26:08Z)
• A Survey of Implicit Discourse Relation Recognition [9.57170901247685]
  implicit discourse relation recognition (IDRR) is to detect implicit relation and classify its sense between two text segments without a connective. This article provides a comprehensive and up-to-date survey for the IDRR task.
  arXiv  Detail & Related papers  (2022-03-06T15:12:53Z)
• GATE: Graph Attention Transformer Encoder for Cross-lingual Relation and Event Extraction [107.8262586956778]
  We introduce graph convolutional networks (GCNs) with universal dependency parses to learn language-agnostic sentence representations. GCNs struggle to model words with long-range dependencies or are not directly connected in the dependency tree. We propose to utilize the self-attention mechanism to learn the dependencies between words with different syntactic distances.
  arXiv  Detail & Related papers  (2020-10-06T20:30:35Z)
• Paragraph-level Commonsense Transformers with Recurrent Memory [77.4133779538797]
  We train a discourse-aware model that incorporates paragraph-level information to generate coherent commonsense inferences from narratives. Our results show that PARA-COMET outperforms the sentence-level baselines, particularly in generating inferences that are both coherent and novel.
  arXiv  Detail & Related papers  (2020-10-04T05:24:12Z)
• Is Supervised Syntactic Parsing Beneficial for Language Understanding? An Empirical Investigation [71.70562795158625]
  Traditional NLP has long held (supervised) syntactic parsing necessary for successful higher-level semantic language understanding (LU) Recent advent of end-to-end neural models, self-supervised via language modeling (LM), and their success on a wide range of LU tasks, questions this belief. We empirically investigate the usefulness of supervised parsing for semantic LU in the context of LM-pretrained transformer networks.
  arXiv  Detail & Related papers  (2020-08-15T21:03:36Z)
• CS-NLP team at SemEval-2020 Task 4: Evaluation of State-of-the-art NLP Deep Learning Architectures on Commonsense Reasoning Task [3.058685580689605]
  We describe our attempt at SemEval-2020 Task 4 competition: Commonsense Validation and Explanation (ComVE) challenge. Our system uses prepared labeled textual datasets that were manually curated for three different natural language inference subtasks. For the second subtask, which is to select the reason why a statement does not make sense, we stand within the first six teams (93.7%) among 27 participants with very competitive results.
  arXiv  Detail & Related papers  (2020-05-17T13:20:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.