Prophet Attention: Predicting Attention with Future Attention for Image Captioning

• URL: http://arxiv.org/abs/2210.10914v2
• Date: Tue, 11 Apr 2023 06:12:01 GMT
• Title: Prophet Attention: Predicting Attention with Future Attention for Image Captioning
• Abstract summary: We propose the Prophet Attention, similar to the form of self-supervision. The proposed Prophet Attention can be easily incorporated into existing image captioning models.
• Score: 99.88870695151874
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, attention based models have been used extensively in many sequence-to-sequence learning systems. Especially for image captioning, the attention based models are expected to ground correct image regions with proper generated words. However, for each time step in the decoding process, the attention based models usually use the hidden state of the current input to attend to the image regions. Under this setting, these attention models have a "deviated focus" problem that they calculate the attention weights based on previous words instead of the one to be generated, impairing the performance of both grounding and captioning. In this paper, we propose the Prophet Attention, similar to the form of self-supervision. In the training stage, this module utilizes the future information to calculate the "ideal" attention weights towards image regions. These calculated "ideal" weights are further used to regularize the "deviated" attention. In this manner, image regions are grounded with the correct words. The proposed Prophet Attention can be easily incorporated into existing image captioning models to improve their performance of both grounding and captioning. The experiments on the Flickr30k Entities and the MSCOCO datasets show that the proposed Prophet Attention consistently outperforms baselines in both automatic metrics and human evaluations. It is worth noticing that we set new state-of-the-arts on the two benchmark datasets and achieve the 1st place on the leaderboard of the online MSCOCO benchmark in terms of the default ranking score, i.e., CIDEr-c40.

Related papers

• Beyond RNNs: Benchmarking Attention-Based Image Captioning Models [0.0]
  This study benchmarks the performance of attention-based image captioning models against RNN-based approaches. We evaluate the effectiveness of Bahdanau attention in enhancing the alignment between image features and generated captions. Our results show that attention-based models outperform RNNs in generating more accurate and semantically rich captions.
  arXiv  Detail & Related papers  (2025-02-26T01:05:18Z)
• Image Captioners Are Scalable Vision Learners Too [61.98796478791261]
  Contrastive pretraining on image-text pairs from the web is one of the most popular large-scale pretraining strategies for vision backbones. Our results show that plain image captioning is a more powerful pretraining strategy than was previously believed.
  arXiv  Detail & Related papers  (2023-06-13T17:18:01Z)
• Correlational Image Modeling for Self-Supervised Visual Pre-Training [81.82907503764775]
  Correlational Image Modeling is a novel and surprisingly effective approach to self-supervised visual pre-training. Three key designs enable correlational image modeling as a nontrivial and meaningful self-supervisory task.
  arXiv  Detail & Related papers  (2023-03-22T15:48:23Z)
• Prompt-based Learning for Unpaired Image Captioning [86.44188293709307]
  Unpaired Image Captioning (UIC) has been developed to learn image descriptions from unaligned vision-language sample pairs. Recent successes of Vision-Language Pre-Trained Models (VL-PTMs) have triggered the development of prompt-based learning. We present in this paper a novel scheme based on prompt to train the UIC model, making best use of the powerful generalization ability.
  arXiv  Detail & Related papers  (2022-05-26T03:13:43Z)
• Supervised Attention in Sequence-to-Sequence Models for Speech Recognition [15.44609280660252]
  We show that attention weights produced by models trained end to end do not always correspond well with actual alignments. We treat the correspondence between attention weights and alignments as a learning problem by imposing a supervised attention loss. Experiments have shown significant improved performance, suggesting that learning the alignments well during training critically determines the performance of sequence-to-sequence models.
  arXiv  Detail & Related papers  (2022-04-25T15:38:48Z)
• Consensus Graph Representation Learning for Better Grounded Image Captioning [48.208119537050166]
  We propose the Consensus Rraph Representation Learning framework (CGRL) for grounded image captioning. We validate the effectiveness of our model, with a significant decline in object hallucination (-9% CHAIRi) on the Flickr30k Entities dataset.
  arXiv  Detail & Related papers  (2021-12-02T04:17:01Z)
• NEAT: Neural Attention Fields for End-to-End Autonomous Driving [59.60483620730437]
  We present NEural ATtention fields (NEAT), a novel representation that enables efficient reasoning for imitation learning models. NEAT is a continuous function which maps locations in Bird's Eye View (BEV) scene coordinates to waypoints and semantics. In a new evaluation setting involving adverse environmental conditions and challenging scenarios, NEAT outperforms several strong baselines and achieves driving scores on par with the privileged CARLA expert.
  arXiv  Detail & Related papers  (2021-09-09T17:55:28Z)
• Dual Attention on Pyramid Feature Maps for Image Captioning [11.372662279301522]
  We propose to apply dual attention on pyramid image feature maps to explore the visual-semantic correlations and improve the quality of generated sentences. We conducted comprehensive experiments on three well-known datasets: Flickr8K, Flickr30K and MS COCO. Our composite captioning model achieves very promising performance in a single-model mode.
  arXiv  Detail & Related papers  (2020-11-02T23:42:34Z)
• Boost Image Captioning with Knowledge Reasoning [10.733743535624509]
  We propose word attention to improve the correctness of visual attention when generating sequential descriptions word-by-word. We introduce a new strategy to inject external knowledge extracted from knowledge graph into the encoder-decoder framework to facilitate meaningful captioning.
  arXiv  Detail & Related papers  (2020-11-02T12:19:46Z)
• Neural Twins Talk [0.0]
  We introduce a novel twin cascaded attention model that outperforms a state-of-the-art image captioning model. Visual grounding ensures the existence of words in the caption sentence that are grounded into a particular region in the input image. We report the results of our experiments in three image captioning tasks on the COCO dataset.
  arXiv  Detail & Related papers  (2020-09-26T06:58:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.