Overview and Insights from the SciVer Shared Task on Scientific Claim Verification

• URL: http://arxiv.org/abs/2107.08188v1
• Date: Sat, 17 Jul 2021 05:47:57 GMT
• Title: Overview and Insights from the SciVer Shared Task on Scientific Claim Verification
• Authors: David Wadden, Kyle Lo
• Abstract summary: We present an overview of the SciVer shared task, presented at the 2nd Scholarly Document Processing (SDP) workshop at NAACL 2021. 11 teams made a total of 14 submissions to the shared task leaderboard, leading to an improvement of more than +23 F1 on the primary task evaluation metric.
• Score: 5.78530472626281
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present an overview of the SciVer shared task, presented at the 2nd Scholarly Document Processing (SDP) workshop at NAACL 2021. In this shared task, systems were provided a scientific claim and a corpus of research abstracts, and asked to identify which articles SUPPORT or REFUTE the claim as well as provide evidentiary sentences justifying those labels. 11 teams made a total of 14 submissions to the shared task leaderboard, leading to an improvement of more than +23 F1 on the primary task evaluation metric. In addition to surveying the participating systems, we provide several insights into modeling approaches to support continued progress and future research on the important and challenging task of scientific claim verification.

Related papers

• Overview of the BioLaySumm 2024 Shared Task on the Lay Summarization of Biomedical Research Articles [21.856049605149646]
  This paper presents the setup and results of the second edition of the BioLaySumm shared task on the Lay Summarisation of Biomedical Research Articles. We aim to build on the first edition's success by further increasing research interest in this important task and encouraging participants to explore novel approaches. Overall, our results show that a broad range of innovative approaches were adopted by task participants, with a predictable shift towards the use of Large Language Models (LLMs)
  arXiv  Detail & Related papers  (2024-08-16T07:00:08Z)
• ArAIEval Shared Task: Propagandistic Techniques Detection in Unimodal and Multimodal Arabic Content [9.287041393988485]
  We present an overview of the second edition of the ArAIEval shared task, organized as part of the Arabic 2024 conference co-located with ACL 2024. In this edition, ArAIEval offers two tasks: (i) detection of propagandistic textual spans with persuasion techniques identification in tweets and news articles, and (ii) distinguishing between propagandistic and non-propagandistic memes. A total of 14 teams participated in the final evaluation phase, with 6 and 9 teams participating in Tasks 1 and 2, respectively.
  arXiv  Detail & Related papers  (2024-07-05T04:28:46Z)
• Overview of the PromptCBLUE Shared Task in CHIP2023 [26.56584015791646]
  This paper presents an overview of the PromptC BLUE shared task held in the CHIP-2023 Conference. It provides a good testbed for Chinese open-domain or medical-domain large language models (LLMs) in general medical natural language processing. This paper describes the tasks, the datasets, evaluation metrics, and the top systems for both tasks.
  arXiv  Detail & Related papers  (2023-12-29T09:05:00Z)
• ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Text [41.3267575540348]
  We present an overview of the ArAIEval shared task, organized as part of the first Arabic 2023 conference co-located with EMNLP 2023. ArAIEval offers two tasks over Arabic text: (i) persuasion technique detection, focusing on identifying persuasion techniques in tweets and news articles, and (ii) disinformation detection in binary and multiclass setups over tweets. A total of 20 teams participated in the final evaluation phase, with 14 and 16 teams participating in Tasks 1 and 2, respectively.
  arXiv  Detail & Related papers  (2023-11-06T15:21:19Z)
• Overview of the BioLaySumm 2023 Shared Task on Lay Summarization of Biomedical Research Articles [47.04555835353173]
  This paper presents the results of the shared task on Lay Summarisation of Biomedical Research Articles (BioLaySumm) hosted at the BioNLP Workshop at ACL 2023. The goal of this shared task is to develop abstractive summarisation models capable of generating "lay summaries" In addition to overall results, we report on the setup and insights from the BioLaySumm shared task, which attracted a total of 20 participating teams across both subtasks.
  arXiv  Detail & Related papers  (2023-09-29T15:43:42Z)
• SciRepEval: A Multi-Format Benchmark for Scientific Document Representations [52.01865318382197]
  We introduce SciRepEval, the first comprehensive benchmark for training and evaluating scientific document representations. We show how state-of-the-art models like SPECTER and SciNCL struggle to generalize across the task formats. A new approach that learns multiple embeddings per document, each tailored to a different format, can improve performance.
  arXiv  Detail & Related papers  (2022-11-23T21:25:39Z)
• IAM: A Comprehensive and Large-Scale Dataset for Integrated Argument Mining Tasks [59.457948080207174]
  In this work, we introduce a comprehensive and large dataset named IAM, which can be applied to a series of argument mining tasks. Near 70k sentences in the dataset are fully annotated based on their argument properties. We propose two new integrated argument mining tasks associated with the debate preparation process: (1) claim extraction with stance classification (CESC) and (2) claim-evidence pair extraction (CEPE)
  arXiv  Detail & Related papers  (2022-03-23T08:07:32Z)
• ICDAR 2021 Competition on Components Segmentation Task of Document Photos [63.289361617237944]
  Three challenge tasks were proposed entailing different segmentation assignments to be performed on a provided dataset. The collected data are from several types of Brazilian ID documents, whose personal information was conveniently replaced. Different Deep Learning models were applied by the entrants with diverse strategies to achieve the best results in each of the tasks.
  arXiv  Detail & Related papers  (2021-06-16T00:49:58Z)
• CAiRE-COVID: A Question Answering and Query-focused Multi-Document Summarization System for COVID-19 Scholarly Information Management [48.251211691263514]
  We present CAiRE-COVID, a real-time question answering (QA) and multi-document summarization system, which won one of the 10 tasks in the Kaggle COVID-19 Open Research dataset Challenge. Our system aims to tackle the recent challenge of mining the numerous scientific articles being published on COVID-19 by answering high priority questions from the community.
  arXiv  Detail & Related papers  (2020-05-04T15:07:27Z)
• Explaining Relationships Between Scientific Documents [55.23390424044378]
  We address the task of explaining relationships between two scientific documents using natural language text. In this paper we establish a dataset of 622K examples from 154K documents.
  arXiv  Detail & Related papers  (2020-02-02T03:54:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.