LookupForensics: A Large-Scale Multi-Task Dataset for Multi-Phase Image-Based Fact Verification

• URL: http://arxiv.org/abs/2407.18614v1
• Date: Fri, 26 Jul 2024 09:15:29 GMT
• Title: LookupForensics: A Large-Scale Multi-Task Dataset for Multi-Phase Image-Based Fact Verification
• Authors: Shuhan Cui, Huy H. Nguyen, Trung-Nghia Le, Chun-Shien Lu, Isao Echizen,
• Abstract summary: We call this "image-based automated fact verification," a name that originated from a text-based fact-checking system used by journalists. We present a large-scale dataset tailored for this new task that features various hand-crafted image edits and machine learning-driven manipulations.
• Score: 15.616232457341097
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Amid the proliferation of forged images, notably the tsunami of deepfake content, extensive research has been conducted on using artificial intelligence (AI) to identify forged content in the face of continuing advancements in counterfeiting technologies. We have investigated the use of AI to provide the original authentic image after deepfake detection, which we believe is a reliable and persuasive solution. We call this "image-based automated fact verification," a name that originated from a text-based fact-checking system used by journalists. We have developed a two-phase open framework that integrates detection and retrieval components. Additionally, inspired by a dataset proposed by Meta Fundamental AI Research, we further constructed a large-scale dataset that is specifically designed for this task. This dataset simulates real-world conditions and includes both content-preserving and content-aware manipulations that present a range of difficulty levels and have potential for ongoing research. This multi-task dataset is fully annotated, enabling it to be utilized for sub-tasks within the forgery identification and fact retrieval domains. This paper makes two main contributions: (1) We introduce a new task, "image-based automated fact verification," and present a novel two-phase open framework combining "forgery identification" and "fact retrieval." (2) We present a large-scale dataset tailored for this new task that features various hand-crafted image edits and machine learning-driven manipulations, with extensive annotations suitable for various sub-tasks. Extensive experimental results validate its practicality for fact verification research and clarify its difficulty levels for various sub-tasks.

Related papers

• Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach [56.55633052479446]
  Web-scale visual entity recognition presents significant challenges due to the lack of clean, large-scale training data. We propose a novel methodology to curate such a dataset, leveraging a multimodal large language model (LLM) for label verification, metadata generation, and rationale explanation. Experiments demonstrate that models trained on this automatically curated data achieve state-of-the-art performance on web-scale visual entity recognition tasks.
  arXiv  Detail & Related papers  (2024-10-31T06:55:24Z)
• VEGA: Learning Interleaved Image-Text Comprehension in Vision-Language Large Models [76.94378391979228]
  We introduce a new, more demanding task known as Interleaved Image-Text (IITC) This task challenges models to discern and disregard superfluous elements in both images and text to accurately answer questions. In support of this task, we further craft a new VEGA dataset, tailored for the IITC task on scientific content, and devised a subtask, Image-Text Association (ITA)
  arXiv  Detail & Related papers  (2024-06-14T17:59:40Z)
• Transformer based Multitask Learning for Image Captioning and Object Detection [13.340784876489927]
  This work introduces a novel multitask learning framework that combines image captioning and object detection into a joint model. We propose TICOD, Transformer-based Image Captioning and Object detection model for jointly training both tasks. Our model outperforms the baselines from image captioning literature by achieving a 3.65% improvement in BERTScore.
  arXiv  Detail & Related papers  (2024-03-10T19:31:13Z)
• Contrastive Transformer Learning with Proximity Data Generation for Text-Based Person Search [60.626459715780605]
  Given a descriptive text query, text-based person search aims to retrieve the best-matched target person from an image gallery. Such a cross-modal retrieval task is quite challenging due to significant modality gap, fine-grained differences and insufficiency of annotated data. In this paper, we propose a simple yet effective dual Transformer model for text-based person search.
  arXiv  Detail & Related papers  (2023-11-15T16:26:49Z)
• Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding [52.723297744257536]
  Pre-trained language models (LMs) have shown effectiveness in scientific literature understanding tasks. We propose a multi-task contrastive learning framework, SciMult, to facilitate common knowledge sharing across different literature understanding tasks.
  arXiv  Detail & Related papers  (2023-05-23T16:47:22Z)
• Self-Supervised Graph Neural Network for Multi-Source Domain Adaptation [51.21190751266442]
  Domain adaptation (DA) tries to tackle the scenarios when the test data does not fully follow the same distribution of the training data. By learning from large-scale unlabeled samples, self-supervised learning has now become a new trend in deep learning. We propose a novel textbfSelf-textbfSupervised textbfGraph Neural Network (SSG) to enable more effective inter-task information exchange and knowledge sharing.
  arXiv  Detail & Related papers  (2022-04-08T03:37:56Z)
• Learning Co-segmentation by Segment Swapping for Retrieval and Discovery [67.6609943904996]
  The goal of this work is to efficiently identify visually similar patterns from a pair of images. We generate synthetic training pairs by selecting object segments in an image and copy-pasting them into another image. We show our approach provides clear improvements for artwork details retrieval on the Brueghel dataset.
  arXiv  Detail & Related papers  (2021-10-29T16:51:16Z)
• Towards Efficient Cross-Modal Visual Textual Retrieval using Transformer-Encoder Deep Features [10.163477961551592]
  Cross-modal retrieval is an important functionality in modern search engines. In this paper, we focus on the image-sentence retrieval task. We use the recently introduced TERN architecture as an image-sentence features extractor.
  arXiv  Detail & Related papers  (2021-06-01T10:11:46Z)
• Exploring Driving-aware Salient Object Detection via Knowledge Transfer [16.105754712355274]
  We construct a driving task-oriented dataset where pixel-level masks of salient objects have been annotated. Cross-domain knowledge difference and task-specific scene gap are two main challenges to focus the salient objects when driving. Inspired by these findings, we proposed a baseline model for the driving task-aware SOD via a knowledge transfer convolutional neural network.
  arXiv  Detail & Related papers  (2021-05-18T05:24:21Z)
• Transformer Reasoning Network for Image-Text Matching and Retrieval [14.238818604272751]
  We consider the problem of accurate image-text matching for the task of multi-modal large-scale information retrieval. We introduce the Transformer Reasoning Network (TERN), an architecture built upon one of the modern relationship-aware self-attentive, the Transformer. TERN is able to separately reason on the two different modalities and to enforce a final common abstract concept space.
  arXiv  Detail & Related papers  (2020-04-20T09:09:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.