The CLEF-2025 CheckThat! Lab: Subjectivity, Fact-Checking, Claim Normalization, and Retrieval

• URL: http://arxiv.org/abs/2503.14828v1
• Date: Wed, 19 Mar 2025 02:06:07 GMT
• Title: The CLEF-2025 CheckThat! Lab: Subjectivity, Fact-Checking, Claim Normalization, and Retrieval
• Authors: Firoj Alam, Julia Maria Struß, Tanmoy Chakraborty, Stefan Dietze, Salim Hafid, Katerina Korre, Arianna Muti, Preslav Nakov, Federico Ruggeri, Sebastian Schellhammer, Vinay Setty, Megha Sundriyal, Konstantin Todorov, Venktesh V,
• Abstract summary: CheckThat! lab aims to advance the development of technologies designed to identify and counteract online disinformation.<n>Since the 2023 edition, the lab has expanded its scope to address auxiliary tasks that support research and decision-making in verification.<n>In the 2025 edition, the lab revisits core verification tasks while also considering auxiliary challenges.
• Score: 47.46368856874347
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The CheckThat! lab aims to advance the development of innovative technologies designed to identify and counteract online disinformation and manipulation efforts across various languages and platforms. The first five editions focused on key tasks in the information verification pipeline, including check-worthiness, evidence retrieval and pairing, and verification. Since the 2023 edition, the lab has expanded its scope to address auxiliary tasks that support research and decision-making in verification. In the 2025 edition, the lab revisits core verification tasks while also considering auxiliary challenges. Task 1 focuses on the identification of subjectivity (a follow-up from CheckThat! 2024), Task 2 addresses claim normalization, Task 3 targets fact-checking numerical claims, and Task 4 explores scientific web discourse processing. These tasks present challenging classification and retrieval problems at both the document and span levels, including multilingual settings.

Related papers

• SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories [55.161075901665946]
  Super aims to capture the realistic challenges faced by researchers working with Machine Learning (ML) and Natural Language Processing (NLP) research repositories. Our benchmark comprises three distinct problem sets: 45 end-to-end problems with annotated expert solutions, 152 sub problems derived from the expert set that focus on specific challenges, and 602 automatically generated problems for larger-scale development. We show that state-of-the-art approaches struggle to solve these problems with the best model (GPT-4o) solving only 16.3% of the end-to-end set, and 46.1% of the scenarios.
  arXiv  Detail & Related papers  (2024-09-11T17:37:48Z)
• V3Det Challenge 2024 on Vast Vocabulary and Open Vocabulary Object Detection: Methods and Results [142.5704093410454]
  The V3Det Challenge 2024 aims to push the boundaries of object detection research. The challenge consists of two tracks: Vast Vocabulary Object Detection and Open Vocabulary Object Detection. We aim to inspire future research directions in vast vocabulary and open-vocabulary object detection.
  arXiv  Detail & Related papers  (2024-06-17T16:58:51Z)
• Perception Test 2023: A Summary of the First Challenge And Outcome [67.0525378209708]
  The First Perception Test challenge was held as a half-day workshop alongside the IEEE/CVF International Conference on Computer Vision (ICCV) 2023. The goal was to benchmarking state-of-the-art video models on the recently proposed Perception Test benchmark. We summarise in this report the task descriptions, metrics, baselines, and results.
  arXiv  Detail & Related papers  (2023-12-20T15:12:27Z)
• Check-worthy Claim Detection across Topics for Automated Fact-checking [21.723689314962233]
  We assess and quantify the challenge of detecting check-worthy claims for new, unseen topics. We propose the AraCWA model to mitigate the performance deterioration when detecting check-worthy claims across topics.
  arXiv  Detail & Related papers  (2022-12-16T14:54:56Z)
• Multimedia Generative Script Learning for Task Planning [58.73725388387305]
  We propose a new task, Multimedia Generative Script Learning, to generate subsequent steps by tracking historical states in both text and vision modalities. This task is challenging in three aspects: the multimedia challenge of capturing the visual states in images, the induction challenge of performing unseen tasks, and the diversity challenge of covering different information in individual steps. Experiment results demonstrate that our approach significantly outperforms strong baselines.
  arXiv  Detail & Related papers  (2022-08-25T19:04:28Z)
• DialFact: A Benchmark for Fact-Checking in Dialogue [56.63709206232572]
  We construct DialFact, a benchmark dataset of 22,245 annotated conversational claims, paired with pieces of evidence from Wikipedia. We find that existing fact-checking models trained on non-dialogue data like FEVER fail to perform well on our task. We propose a simple yet data-efficient solution to effectively improve fact-checking performance in dialogue.
  arXiv  Detail & Related papers  (2021-10-15T17:34:35Z)
• UPV at CheckThat! 2021: Mitigating Cultural Differences for Identifying Multilingual Check-worthy Claims [6.167830237917659]
  In this paper, we propose a language identification task as an auxiliary task to mitigate unintended bias. Our results show that joint training of language identification and check-worthy claim detection tasks can provide performance gains for some of the selected languages.
  arXiv  Detail & Related papers  (2021-09-19T21:46:16Z)
• Overview and Insights from the SciVer Shared Task on Scientific Claim Verification [5.78530472626281]
  We present an overview of the SciVer shared task, presented at the 2nd Scholarly Document Processing (SDP) workshop at NAACL 2021. 11 teams made a total of 14 submissions to the shared task leaderboard, leading to an improvement of more than +23 F1 on the primary task evaluation metric.
  arXiv  Detail & Related papers  (2021-07-17T05:47:57Z)
• A Review on Fact Extraction and Verification [19.373340472113703]
  We study the fact checking problem, which aims to identify the veracity of a given claim. We focus on the task of Fact Extraction and VERification (FEVER) and its accompanied dataset. This task is essential and can be the building block of applications such as fake news detection and medical claim verification.
  arXiv  Detail & Related papers  (2020-10-06T20:05:43Z)
• Overview of CheckThat! 2020: Automatic Identification and Verification of Claims in Social Media [26.60148306714383]
  We present an overview of the third edition of the CheckThat! Lab at CLEF 2020. The lab featured five tasks in two different languages: English and Arabic. We describe the tasks setup, the evaluation results, and a summary of the approaches used by the participants.
  arXiv  Detail & Related papers  (2020-07-15T21:19:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.