Large Language Models and Provenance Metadata for Determining the Relevance of Images and Videos in News Stories

• URL: http://arxiv.org/abs/2502.09689v1
• Date: Thu, 13 Feb 2025 16:48:27 GMT
• Title: Large Language Models and Provenance Metadata for Determining the Relevance of Images and Videos in News Stories
• Authors: Tomas Peterka, Matyas Bohacek,
• Abstract summary: multimodal misinformation campaigns often combine text with images and videos taken out of context. We propose a system that analyzes both the article's text and the provenance metadata of included images and videos to determine whether they are relevant. We open-source the system prototype and interactive web interface.
• Score: 0.7366405857677227
• License:
• Abstract: The most effective misinformation campaigns are multimodal, often combining text with images and videos taken out of context -- or fabricating them entirely -- to support a given narrative. Contemporary methods for detecting misinformation, whether in deepfakes or text articles, often miss the interplay between multiple modalities. Built around a large language model, the system proposed in this paper addresses these challenges. It analyzes both the article's text and the provenance metadata of included images and videos to determine whether they are relevant. We open-source the system prototype and interactive web interface.

Related papers

• Multi-Modal interpretable automatic video captioning [1.9874264019909988]
  We introduce a novel video captioning method trained with multi-modal contrastive loss. Our approach is designed to capture the dependency between these modalities, resulting in more accurate, thus pertinent captions.
  arXiv  Detail & Related papers  (2024-11-11T11:12:23Z)
• Target Prompting for Information Extraction with Vision Language Model [0.0]
  The recent trend in the Large Vision and Language model has brought a new change in how information extraction systems are built. They are significantly better at generating text from document images and providing accurate answers to questions. General prompting techniques used with large language models are often not suitable for these specially designed vision language models.
  arXiv  Detail & Related papers  (2024-08-07T15:17:51Z)
• TRINS: Towards Multimodal Language Models that Can Read [61.17806538631744]
  TRINS is a Text-Rich image INStruction dataset. It contains 39,153 text-rich images, captions, and 102,437 questions. We introduce a Language-vision Reading Assistant (LaRA) which is good at understanding textual content within images.
  arXiv  Detail & Related papers  (2024-06-10T18:52:37Z)
• Interpretable Detection of Out-of-Context Misinformation with Neural-Symbolic-Enhanced Large Multimodal Model [16.348950072491697]
  Misinformation creators now more tend to use out-of- multimedia contents to deceive the public and fake news detection systems. This new type of misinformation increases the difficulty of not only detection but also clarification, because every individual modality is close enough to true information. In this paper we explore how to achieve interpretable cross-modal de-contextualization detection that simultaneously identifies the mismatched pairs and the cross-modal contradictions.
  arXiv  Detail & Related papers  (2023-04-15T21:11:55Z)
• NewsStories: Illustrating articles with visual summaries [49.924916589209374]
  We introduce a large-scale multimodal dataset containing over 31M articles, 22M images and 1M videos. We show that state-of-the-art image-text alignment methods are not robust to longer narratives with multiple images. We introduce an intuitive baseline that outperforms these methods on zero-shot image-set retrieval by 10% on the GoodNews dataset.
  arXiv  Detail & Related papers  (2022-07-26T17:34:11Z)
• Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrieval [70.30052749168013]
  Multi-channel video-language retrieval require models to understand information from different channels. contrastive multimodal models are shown to be highly effective at aligning entities in images/videos and text. There is not a clear way to quickly adapt these two lines to multi-channel video-language retrieval with limited data and resources.
  arXiv  Detail & Related papers  (2022-06-05T01:43:52Z)
• Visually-Augmented Language Modeling [137.36789885105642]
  We propose a novel pre-training framework, named VaLM, to Visually-augment text tokens with retrieved relevant images for Language Modeling. With the visually-augmented context, VaLM uses a visual knowledge fusion layer to enable multimodal grounded language modeling. We evaluate the proposed model on various multimodal commonsense reasoning tasks, which require visual information to excel.
  arXiv  Detail & Related papers  (2022-05-20T13:41:12Z)
• NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media [93.51739200834837]
  We propose a dataset where both image and text are unmanipulated but mismatched. We introduce several strategies for automatic retrieval of suitable images for the given captions. Our large-scale automatically generated NewsCLIPpings dataset requires models to jointly analyze both modalities.
  arXiv  Detail & Related papers  (2021-04-13T01:53:26Z)
• On the Role of Images for Analyzing Claims in Social Media [3.8142537449670963]
  We present an empirical study on visual, textual, and multimodal models for the tasks of claim, claim check-worthiness, and conspiracy detection. Recent work suggests that images are more influential than text and often appear alongside fake text.
  arXiv  Detail & Related papers  (2021-03-17T12:40:27Z)
• Cross-Media Keyphrase Prediction: A Unified Framework with Multi-Modality Multi-Head Attention and Image Wordings [63.79979145520512]
  We explore the joint effects of texts and images in predicting the keyphrases for a multimedia post. We propose a novel Multi-Modality Multi-Head Attention (M3H-Att) to capture the intricate cross-media interactions. Our model significantly outperforms the previous state of the art based on traditional attention networks.
  arXiv  Detail & Related papers  (2020-11-03T08:44:18Z)
• Multimodal Analytics for Real-world News using Measures of Cross-modal Entity Consistency [8.401772200450417]
  Multimodal information, e.g., enriching text with photos, is typically used to convey the news more effectively or to attract attention. We introduce a novel task of cross-modal consistency verification in real-world news and present a multimodal approach to quantify the entity coherence between image and text.
  arXiv  Detail & Related papers  (2020-03-23T17:49:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.