SemEval-2021 Task 4: Reading Comprehension of Abstract Meaning

• URL: http://arxiv.org/abs/2105.14879v2
• Date: Tue, 1 Jun 2021 10:45:27 GMT
• Title: SemEval-2021 Task 4: Reading Comprehension of Abstract Meaning
• Authors: Boyuan Zheng, Xiaoyu Yang, Yu-Ping Ruan, Zhenhua Ling, Quan Liu, Si Wei, Xiaodan Zhu
• Abstract summary: This paper introduces the SemEval-2021 shared task 4: Reading of Abstract Meaning (ReCAM) Given a passage and the corresponding question, a participating system is expected to choose the correct answer from five candidates of abstract concepts. Subtask 1 aims to evaluate how well a system can model concepts that cannot be directly perceived in the physical world. Subtask 2 focuses on models' ability in comprehending nonspecific concepts located high in a hypernym hierarchy. Subtask 3 aims to provide some insights into models' generalizability over the two types of abstractness.
• Score: 47.49596196559958
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper introduces the SemEval-2021 shared task 4: Reading Comprehension of Abstract Meaning (ReCAM). This shared task is designed to help evaluate the ability of machines in representing and understanding abstract concepts. Given a passage and the corresponding question, a participating system is expected to choose the correct answer from five candidates of abstract concepts in a cloze-style machine reading comprehension setup. Based on two typical definitions of abstractness, i.e., the imperceptibility and nonspecificity, our task provides three subtasks to evaluate the participating models. Specifically, Subtask 1 aims to evaluate how well a system can model concepts that cannot be directly perceived in the physical world. Subtask 2 focuses on models' ability in comprehending nonspecific concepts located high in a hypernym hierarchy given the context of a passage. Subtask 3 aims to provide some insights into models' generalizability over the two types of abstractness. During the SemEval-2021 official evaluation period, we received 23 submissions to Subtask 1 and 28 to Subtask 2. The participating teams additionally made 29 submissions to Subtask 3. The leaderboard and competition website can be found at https://competitions.codalab.org/competitions/26153. The data and baseline code are available at https://github.com/boyuanzheng010/SemEval2021-Reading-Comprehension-of-Abstract-Meaning.

Related papers

• Fair Abstractive Summarization of Diverse Perspectives [103.08300574459783]
  A fair summary should provide a comprehensive coverage of diverse perspectives without underrepresenting certain groups. We first formally define fairness in abstractive summarization as not underrepresenting perspectives of any groups of people. We propose four reference-free automatic metrics by measuring the differences between target and source perspectives.
  arXiv  Detail & Related papers  (2023-11-14T03:38:55Z)
• Overview of the BioLaySumm 2023 Shared Task on Lay Summarization of Biomedical Research Articles [47.04555835353173]
  This paper presents the results of the shared task on Lay Summarisation of Biomedical Research Articles (BioLaySumm) hosted at the BioNLP Workshop at ACL 2023. The goal of this shared task is to develop abstractive summarisation models capable of generating "lay summaries" In addition to overall results, we report on the setup and insights from the BioLaySumm shared task, which attracted a total of 20 participating teams across both subtasks.
  arXiv  Detail & Related papers  (2023-09-29T15:43:42Z)
• A Unified Framework for Multi-intent Spoken Language Understanding with prompting [14.17726194025463]
  We describe a Prompt-based Spoken Language Understanding (PromptSLU) framework, to intuitively unify two sub-tasks into the same form. In detail, ID and SF are completed by concisely filling the utterance into task-specific prompt templates as input, and sharing output formats of key-value pairs sequence. Experiment results show that our framework outperforms several state-of-the-art baselines on two public datasets.
  arXiv  Detail & Related papers  (2022-10-07T05:58:05Z)
• IIE-NLP-Eyas at SemEval-2021 Task 4: Enhancing PLM for ReCAM with Special Tokens, Re-Ranking, Siamese Encoders and Back Translation [8.971288666318719]
  This paper introduces our systems for all three subtasks of SemEval-2021 Task 4: Reading of Abstract Meaning. We well-design many simple and effective approaches adapted to the backbone model (RoBERTa) Experimental results show that our approaches achieve significant performance compared with the baseline systems.
  arXiv  Detail & Related papers  (2021-02-25T10:51:48Z)
• What Makes a Good Summary? Reconsidering the Focus of Automatic Summarization [49.600619575148706]
  We find that the current focus of the field does not fully align with participants' wishes. Based on our findings, we argue that it is important to adopt a broader perspective on automatic summarization.
  arXiv  Detail & Related papers  (2020-12-14T15:12:35Z)
• SemEval-2020 Task 5: Counterfactual Recognition [36.38097292055921]
  Subtask-1 aims to determine whether a given sentence is a counterfactual statement or not. Subtask-2 requires the participating systems to extract the antecedent and consequent in a given counterfactual statement.
  arXiv  Detail & Related papers  (2020-08-02T20:32:19Z)
• SemEval-2020 Task 4: Commonsense Validation and Explanation [24.389998904122244]
  SemEval-2020 Task 4, Commonsense Validation and Explanation (ComVE), includes three subtasks. We aim to evaluate whether a system can distinguish a natural language statement that makes sense to humans from one that does not. For Subtask A and Subtask B, the performances of top-ranked systems are close to that of humans.
  arXiv  Detail & Related papers  (2020-07-01T04:41:05Z)
• Document Modeling with Graph Attention Networks for Multi-grained Machine Reading Comprehension [127.3341842928421]
  Natural Questions is a new challenging machine reading comprehension benchmark. It has two-grained answers, which are a long answer (typically a paragraph) and a short answer (one or more entities inside the long answer) Existing methods treat these two sub-tasks individually during training while ignoring their dependencies. We present a novel multi-grained machine reading comprehension framework that focuses on modeling documents at their hierarchical nature.
  arXiv  Detail & Related papers  (2020-05-12T14:20:09Z)
• Exploring Explainable Selection to Control Abstractive Summarization [51.74889133688111]
  We develop a novel framework that focuses on explainability. A novel pair-wise matrix captures the sentence interactions, centrality, and attribute scores. A sentence-deployed attention mechanism in the abstractor ensures the final summary emphasizes the desired content.
  arXiv  Detail & Related papers  (2020-04-24T14:39:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.