LocCa: Visual Pretraining with Location-aware Captioners

• URL: http://arxiv.org/abs/2403.19596v2
• Date: Mon, 11 Nov 2024 22:39:35 GMT
• Title: LocCa: Visual Pretraining with Location-aware Captioners
• Authors: Bo Wan, Michael Tschannen, Yongqin Xian, Filip Pavetic, Ibrahim Alabdulmohsin, Xiao Wang, André Susano Pinto, Andreas Steiner, Lucas Beyer, Xiaohua Zhai,
• Abstract summary: We propose a simple visual pretraining method with location-aware captioners (LocCa) LocCa uses a simple image captioner task interface to teach a model to read out rich information. Thanks to the multitask capabilities of an encoder-decoder architecture, we show that an image captioner can easily handle multiple tasks during pretraining.
• Score: 53.93041020615046
• License:
• Abstract: Image captioning has been shown as an effective pretraining method similar to contrastive pretraining. However, the incorporation of location-aware information into visual pretraining remains an area with limited research. In this paper, we propose a simple visual pretraining method with location-aware captioners (LocCa). LocCa uses a simple image captioner task interface, to teach a model to read out rich information, i.e. bounding box coordinates, and captions, conditioned on the image pixel input. Thanks to the multitask capabilities of an encoder-decoder architecture, we show that an image captioner can easily handle multiple tasks during pretraining. Our experiments demonstrate that LocCa outperforms standard captioners significantly on localization downstream tasks while maintaining comparable performance on holistic tasks.

Related papers

• Image Captioners Are Scalable Vision Learners Too [61.98796478791261]
  Contrastive pretraining on image-text pairs from the web is one of the most popular large-scale pretraining strategies for vision backbones. Our results show that plain image captioning is a more powerful pretraining strategy than was previously believed.
  arXiv  Detail & Related papers  (2023-06-13T17:18:01Z)
• DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training [73.74291217502928]
  We propose a simple framework, named DeCap, for zero-shot captioning. We introduce a lightweight visual-aware language decoder. We project the visual embedding into the CLIP text embedding space, while the projected embedding retains the information of the visual input.
  arXiv  Detail & Related papers  (2023-03-06T11:02:47Z)
• CapDet: Unifying Dense Captioning and Open-World Detection Pretraining [68.8382821890089]
  We propose a novel open-world detector named CapDet to either predict under a given category list or directly generate the category of predicted bounding boxes. Specifically, we unify the open-world detection and dense caption tasks into a single yet effective framework by introducing an additional dense captioning head.
  arXiv  Detail & Related papers  (2023-03-04T19:53:00Z)
• Retrieval-augmented Image Captioning [15.266569206458648]
  We present a new approach to image captioning that generates sentences given the input image and a set of captions retrieved from a datastore. The encoder in our model jointly processes the image and retrieved captions using a pretrained V&L BERT. Our work contributes towards using pretrained V&L encoders for generative tasks, instead of standard classification tasks.
  arXiv  Detail & Related papers  (2023-02-16T12:54:13Z)
• Large-Scale Bidirectional Training for Zero-Shot Image Captioning [44.17587735943739]
  We introduce Bidirectional Image Text Training in largER Scale, BITTERS, an efficient training and inference framework for zero-shot image captioning. We show that careful selection of large-scale training set and model architecture is the key to achieving zero-shot image captioning.
  arXiv  Detail & Related papers  (2022-11-13T00:09:36Z)
• What is Where by Looking: Weakly-Supervised Open-World Phrase-Grounding without Text Inputs [82.93345261434943]
  Given an input image, and nothing else, our method returns the bounding boxes of objects in the image and phrases that describe the objects. This is achieved within an open world paradigm, in which the objects in the input image may not have been encountered during the training of the localization mechanism. Our work generalizes weakly supervised segmentation and phrase grounding and is shown empirically to outperform the state of the art in both domains.
  arXiv  Detail & Related papers  (2022-06-19T09:07:30Z)
• CapOnImage: Context-driven Dense-Captioning on Image [13.604173177437536]
  We introduce a new task called captioning on image (CapOnImage), which aims to generate dense captions at different locations of the image based on contextual information. We propose a multi-modal pre-training model with multi-level pre-training tasks that progressively learn the correspondence between texts and image locations. Compared with other image captioning model variants, our model achieves the best results in both captioning accuracy and diversity aspects.
  arXiv  Detail & Related papers  (2022-04-27T14:40:31Z)
• ClipCap: CLIP Prefix for Image Captioning [6.69087470775851]
  We use CLIP encoding as a prefix to the caption, by employing a simple mapping network, and then fine-tunes a language model to generate the image captions. We demonstrate our model achieves comparable results to state-of-the-art methods on the challenging Conceptual Captions and nocaps datasets.
  arXiv  Detail & Related papers  (2021-11-18T14:49:15Z)
• VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning [128.6138588412508]
  This paper presents VIsual VOcabulary pretraining (VIVO) that performs pre-training in the absence of caption annotations. Our model can not only generate fluent image captions that describe novel objects, but also identify the locations of these objects.
  arXiv  Detail & Related papers  (2020-09-28T23:20:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.