Paraphrase Acquisition from Image Captions
- URL: http://arxiv.org/abs/2301.11030v1
- Date: Thu, 26 Jan 2023 10:54:51 GMT
- Title: Paraphrase Acquisition from Image Captions
- Authors: Marcel Gohsen and Matthias Hagen and Martin Potthast and Benno Stein
- Abstract summary: We propose to use captions from the Web as a previously underutilized resource for paraphrases.
We analyze captions in the English Wikipedia, where editors frequently relabel the same image for different articles.
We introduce characteristic maps along the two similarity dimensions to identify the style of paraphrases coming from different sources.
- Score: 36.94459555199183
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We propose to use captions from the Web as a previously underutilized
resource for paraphrases (i.e., texts with the same "message") and to create
and analyze a corresponding dataset. When an image is reused on the Web, an
original caption is often assigned. We hypothesize that different captions for
the same image naturally form a set of mutual paraphrases. To demonstrate the
suitability of this idea, we analyze captions in the English Wikipedia, where
editors frequently relabel the same image for different articles. The paper
introduces the underlying mining technology and compares known paraphrase
corpora with respect to their syntactic and semantic paraphrase similarity to
our new resource. In this context, we introduce characteristic maps along the
two similarity dimensions to identify the style of paraphrases coming from
different sources. An annotation study demonstrates the high reliability of the
algorithmically determined characteristic maps.
Related papers
- Cross-Domain Image Captioning with Discriminative Finetuning [20.585138136033905]
Fine-tuning an out-of-the-box neural captioner with a self-supervised discriminative communication objective helps to recover a plain, visually descriptive language.
We show that discriminatively finetuned captions are more helpful than either vanilla ClipCap captions or ground-truth captions for human annotators tasked with an image discrimination task.
arXiv Detail & Related papers (2023-04-04T09:33:16Z) - Transform, Contrast and Tell: Coherent Entity-Aware Multi-Image
Captioning [0.65268245109828]
Coherent entity-aware multi-image captioning aims to generate coherent captions for neighboring images in a news document.
This paper proposes a coherent entity-aware multi-image captioning model by making use of coherence relationships.
arXiv Detail & Related papers (2023-02-04T07:50:31Z) - Show, Interpret and Tell: Entity-aware Contextualised Image Captioning
in Wikipedia [10.21762162291523]
We propose the novel task of captioning Wikipedia images by integrating contextual knowledge.
Specifically, we produce models that jointly reason over Wikipedia articles, Wikimedia images and their associated descriptions.
arXiv Detail & Related papers (2022-09-21T16:14:15Z) - The Case for Perspective in Multimodal Datasets [0.9786690381850356]
We present a set of experiments in which FrameNet annotation is applied to the Multi30k and the Flickr 30k Entities datasets.
We assess the cosine similarity between the semantic representations derived from the annotation of both pictures and captions for frames.
arXiv Detail & Related papers (2022-05-22T18:37:05Z) - On Distinctive Image Captioning via Comparing and Reweighting [52.3731631461383]
In this paper, we aim to improve the distinctiveness of image captions via comparing and reweighting with a set of similar images.
Our metric reveals that the human annotations of each image in the MSCOCO dataset are not equivalent based on distinctiveness.
In contrast, previous works normally treat the human annotations equally during training, which could be a reason for generating less distinctive captions.
arXiv Detail & Related papers (2022-04-08T08:59:23Z) - Group-based Distinctive Image Captioning with Memory Attention [45.763534774116856]
Group-based Distinctive Captioning Model (GdisCap) improves the distinctiveness of image captions.
New evaluation metric, distinctive word rate (DisWordRate) is proposed to measure the distinctiveness of captions.
arXiv Detail & Related papers (2021-08-20T12:46:36Z) - NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media [93.51739200834837]
We propose a dataset where both image and text are unmanipulated but mismatched.
We introduce several strategies for automatic retrieval of suitable images for the given captions.
Our large-scale automatically generated NewsCLIPpings dataset requires models to jointly analyze both modalities.
arXiv Detail & Related papers (2021-04-13T01:53:26Z) - Telling the What while Pointing the Where: Fine-grained Mouse Trace and
Language Supervision for Improved Image Retrieval [60.24860627782486]
Fine-grained image retrieval often requires the ability to also express the where in the image the content they are looking for is.
In this paper, we describe an image retrieval setup where the user simultaneously describes an image using both spoken natural language (the "what") and mouse traces over an empty canvas (the "where")
Our model is capable of taking this spatial guidance into account, and provides more accurate retrieval results compared to text-only equivalent systems.
arXiv Detail & Related papers (2021-02-09T17:54:34Z) - Intrinsic Image Captioning Evaluation [53.51379676690971]
We propose a learning based metrics for image captioning, which we call Intrinsic Image Captioning Evaluation(I2CE)
Experiment results show that our proposed method can keep robust performance and give more flexible scores to candidate captions when encountered with semantic similar expression or less aligned semantics.
arXiv Detail & Related papers (2020-12-14T08:36:05Z) - Compare and Reweight: Distinctive Image Captioning Using Similar Images
Sets [52.3731631461383]
We aim to improve the distinctiveness of image captions through training with sets of similar images.
Our metric shows that the human annotations of each image are not equivalent based on distinctiveness.
arXiv Detail & Related papers (2020-07-14T07:40:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.