A TextGCN-Based Decoding Approach for Improving Remote Sensing Image Captioning

• URL: http://arxiv.org/abs/2409.18467v3
• Date: Sat, 12 Oct 2024 04:15:09 GMT
• Title: A TextGCN-Based Decoding Approach for Improving Remote Sensing Image Captioning
• Authors: Swadhin Das, Raksha Sharma,
• Abstract summary: We propose a novel encoder-decoder setup that deploys a Text Graph Convolutional Network (TextGCN) and multi-layer LSTMs. The embeddings generated by TextGCN enhance the decoder's understanding by capturing the semantic relationships among words at both the sentence and corpus levels. We present an extensive evaluation of our approach against various other state-of-the-art encoder-decoder frameworks.
• Score: 0.15346678870160887
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Remote sensing images are highly valued for their ability to address complex real-world issues such as risk management, security, and meteorology. However, manually captioning these images is challenging and requires specialized knowledge across various domains. This letter presents an approach for automatically describing (captioning) remote sensing images. We propose a novel encoder-decoder setup that deploys a Text Graph Convolutional Network (TextGCN) and multi-layer LSTMs. The embeddings generated by TextGCN enhance the decoder's understanding by capturing the semantic relationships among words at both the sentence and corpus levels. Furthermore, we advance our approach with a comparison-based beam search method to ensure fairness in the search strategy for generating the final caption. We present an extensive evaluation of our approach against various other state-of-the-art encoder-decoder frameworks. We evaluated our method across three datasets using seven metrics: BLEU-1 to BLEU-4, METEOR, ROUGE-L, and CIDEr. The results demonstrate that our approach significantly outperforms other state-of-the-art encoder-decoder methods.

Related papers

• Decoder Pre-Training with only Text for Scene Text Recognition [54.93037783663204]
  Scene text recognition (STR) pre-training methods have achieved remarkable progress, primarily relying on synthetic datasets. We introduce a novel method named Decoder Pre-training with only text for STR (DPTR) DPTR treats text embeddings produced by the CLIP text encoder as pseudo visual embeddings and uses them to pre-train the decoder.
  arXiv  Detail & Related papers  (2024-08-11T06:36:42Z)
• Perceptual Image Compression with Cooperative Cross-Modal Side Information [53.356714177243745]
  We propose a novel deep image compression method with text-guided side information to achieve a better rate-perception-distortion tradeoff. Specifically, we employ the CLIP text encoder and an effective Semantic-Spatial Aware block to fuse the text and image features.
  arXiv  Detail & Related papers  (2023-11-23T08:31:11Z)
• Adjacent Context Coordination Network for Salient Object Detection in Optical Remote Sensing Images [102.75699068451166]
  We propose a novel Adjacent Context Coordination Network (ACCoNet) to explore the coordination of adjacent features in an encoder-decoder architecture for optical RSI-SOD. The proposed ACCoNet outperforms 22 state-of-the-art methods under nine evaluation metrics, and runs up to 81 fps on a single NVIDIA Titan X GPU.
  arXiv  Detail & Related papers  (2022-03-25T14:14:55Z)
• CRIS: CLIP-Driven Referring Image Segmentation [71.56466057776086]
  We propose an end-to-end CLIP-Driven Referring Image framework (CRIS) CRIS resorts to vision-language decoding and contrastive learning for achieving the text-to-pixel alignment. Our proposed framework significantly outperforms the state-of-the-art performance without any post-processing.
  arXiv  Detail & Related papers  (2021-11-30T07:29:08Z)
• Empirical Analysis of Image Caption Generation using Deep Learning [0.0]
  We have implemented and experimented with various flavors of multi-modal image captioning networks. The goal is to analyze the performance of each approach using various evaluation metrics.
  arXiv  Detail & Related papers  (2021-05-14T05:38:13Z)
• Exploring Explicit and Implicit Visual Relationships for Image Captioning [11.82805641934772]
  In this paper, we explore explicit and implicit visual relationships to enrich region-level representations for image captioning. Explicitly, we build semantic graph over object pairs and exploit gated graph convolutional networks (Gated GCN) to selectively aggregate local neighbors' information. Implicitly, we draw global interactions among the detected objects through region-based bidirectional encoder representations from transformers.
  arXiv  Detail & Related papers  (2021-05-06T01:47:51Z)
• A Novel Actor Dual-Critic Model for Remote Sensing Image Captioning [32.11006090613004]
  We deal with the problem of generating textual captions from optical remote sensing (RS) images using the notion of deep reinforcement learning. We introduce an Actor Dual-Critic training strategy where a second critic model is deployed in the form of an encoder-decoder RNN. We observe that the proposed model generates sentences on the test data highly similar to the ground truth and is successful in generating even better captions in many critical cases.
  arXiv  Detail & Related papers  (2020-10-05T13:35:02Z)
• DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis [80.54273334640285]
  We propose a novel one-stage text-to-image backbone that directly synthesizes high-resolution images without entanglements between different generators. We also propose a novel Target-Aware Discriminator composed of Matching-Aware Gradient Penalty and One-Way Output. Compared with current state-of-the-art methods, our proposed DF-GAN is simpler but more efficient to synthesize realistic and text-matching images.
  arXiv  Detail & Related papers  (2020-08-13T12:51:17Z)
• Beyond Single Stage Encoder-Decoder Networks: Deep Decoders for Semantic Image Segmentation [56.44853893149365]
  Single encoder-decoder methodologies for semantic segmentation are reaching their peak in terms of segmentation quality and efficiency per number of layers. We propose a new architecture based on a decoder which uses a set of shallow networks for capturing more information content. In order to further improve the architecture we introduce a weight function which aims to re-balance classes to increase the attention of the networks to under-represented objects.
  arXiv  Detail & Related papers  (2020-07-19T18:44:34Z)
• SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognition [17.191496890376197]
  We propose a semantics enhanced encoder-decoder framework to robustly recognize low-quality scene texts. The proposed framework is more robust for low-quality text images, and achieves state-of-the-art results on several benchmark datasets.
  arXiv  Detail & Related papers  (2020-05-22T03:02:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.