Good Representation, Better Explanation: Role of Convolutional Neural Networks in Transformer-Based Remote Sensing Image Captioning

• URL: http://arxiv.org/abs/2502.16095v1
• Date: Sat, 22 Feb 2025 05:36:28 GMT
• Title: Good Representation, Better Explanation: Role of Convolutional Neural Networks in Transformer-Based Remote Sensing Image Captioning
• Authors: Swadhin Das, Saarthak Gupta, and Kamal Kumar, Raksha Sharma,
• Abstract summary: We systematically evaluate twelve different convolutional neural network (CNN) architectures within a transformer-based encoder framework to assess their effectiveness in Remote Sensing Image Captioning (RSIC)<n>The results highlight the critical role of encoder selection in improving captioning performance, demonstrating that specific CNN architectures significantly enhance the quality of generated descriptions for remote sensing images.
• Score: 0.6058427379240696
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Remote Sensing Image Captioning (RSIC) is the process of generating meaningful descriptions from remote sensing images. Recently, it has gained significant attention, with encoder-decoder models serving as the backbone for generating meaningful captions. The encoder extracts essential visual features from the input image, transforming them into a compact representation, while the decoder utilizes this representation to generate coherent textual descriptions. Recently, transformer-based models have gained significant popularity due to their ability to capture long-range dependencies and contextual information. The decoder has been well explored for text generation, whereas the encoder remains relatively unexplored. However, optimizing the encoder is crucial as it directly influences the richness of extracted features, which in turn affects the quality of generated captions. To address this gap, we systematically evaluate twelve different convolutional neural network (CNN) architectures within a transformer-based encoder framework to assess their effectiveness in RSIC. The evaluation consists of two stages: first, a numerical analysis categorizes CNNs into different clusters, based on their performance. The best performing CNNs are then subjected to human evaluation from a human-centric perspective by a human annotator. Additionally, we analyze the impact of different search strategies, namely greedy search and beam search, to ensure the best caption. The results highlight the critical role of encoder selection in improving captioning performance, demonstrating that specific CNN architectures significantly enhance the quality of generated descriptions for remote sensing images. By providing a detailed comparison of multiple encoders, this study offers valuable insights to guide advances in transformer-based image captioning models.

Related papers

• FE-LWS: Refined Image-Text Representations via Decoder Stacking and Fused Encodings for Remote Sensing Image Captioning [0.15346678870160887]
  This paper introduces a novel approach that integrates features from two distinct CNN based encoders.<n>We also propose a weighted averaging technique to combine the outputs of all GRUs in the stacked decoder.<n>The results demonstrate that our fusion-based approach, along with the enhanced stacked decoder, significantly outperforms both the transformer-based state-of-the-art model and other LSTM-based baselines.
  arXiv  Detail & Related papers  (2025-02-13T12:54:13Z)
• Unleashing Text-to-Image Diffusion Prior for Zero-Shot Image Captioning [70.98890307376548]
  We propose a novel Patch-wise Cross-modal feature Mix-up (PCM) mechanism to adaptively mitigate the unfaithful contents during training.<n>Our PCM-Net ranks first in both in-domain and cross-domain zero-shot image captioning.
  arXiv  Detail & Related papers  (2024-12-31T13:39:08Z)
• Compressed Image Captioning using CNN-based Encoder-Decoder Framework [0.0]
  We develop an automatic image captioning architecture that combines the strengths of convolutional neural networks (CNNs) and encoder-decoder models. We also do a performance comparison where we delved into the realm of pre-trained CNN models. In our quest for optimization, we also explored the integration of frequency regularization techniques to compress the "AlexNet" and "EfficientNetB0" models.
  arXiv  Detail & Related papers  (2024-04-28T03:47:48Z)
• An Image captioning algorithm based on the Hybrid Deep Learning Technique (CNN+GRU) [0.0]
  We present a CNN-GRU encoder decode framework for caption-to-image reconstructor. It handles the semantic context into consideration as well as the time complexity. The suggested model outperforms the state-of-the-art LSTM-A5 model for picture captioning in terms of time complexity and accuracy.
  arXiv  Detail & Related papers  (2023-01-06T10:00:06Z)
• Image Captioning based on Feature Refinement and Reflective Decoding [0.0]
  This paper introduces an encoder-decoder-based image captioning system. It extracts spatial and global features for each region in the image using the Faster R-CNN with ResNet-101 as a backbone. The decoder consists of an attention-based recurrent module and a reflective attention module to enhance the decoder's ability to model long-term sequential dependencies.
  arXiv  Detail & Related papers  (2022-06-16T07:56:28Z)
• MaskOCR: Text Recognition with Masked Encoder-Decoder Pretraining [68.05105411320842]
  We propose a novel approach MaskOCR to unify vision and language pre-training in the classical encoder-decoder recognition framework. We adopt the masked image modeling approach to pre-train the feature encoder using a large set of unlabeled real text images. We transform text data into synthesized text images to unify the data modalities of vision and language, and enhance the language modeling capability of the sequence decoder.
  arXiv  Detail & Related papers  (2022-06-01T08:27:19Z)
• Neural Data-Dependent Transform for Learned Image Compression [72.86505042102155]
  We build a neural data-dependent transform and introduce a continuous online mode decision mechanism to jointly optimize the coding efficiency for each individual image. The experimental results show the effectiveness of the proposed neural-syntax design and the continuous online mode decision mechanism.
  arXiv  Detail & Related papers  (2022-03-09T14:56:48Z)
• Injecting Semantic Concepts into End-to-End Image Captioning [61.41154537334627]
  We propose a pure vision transformer-based image captioning model, dubbed as ViTCAP, in which grid representations are used without extracting the regional features. For improved performance, we introduce a novel Concept Token Network (CTN) to predict the semantic concepts and then incorporate them into the end-to-end captioning. In particular, the CTN is built on the basis of a vision transformer and is designed to predict the concept tokens through a classification task.
  arXiv  Detail & Related papers  (2021-12-09T22:05:05Z)
• Dynamic Neural Representational Decoders for High-Resolution Semantic Segmentation [98.05643473345474]
  We propose a novel decoder, termed dynamic neural representational decoder (NRD) As each location on the encoder's output corresponds to a local patch of the semantic labels, in this work, we represent these local patches of labels with compact neural networks. This neural representation enables our decoder to leverage the smoothness prior in the semantic label space, and thus makes our decoder more efficient.
  arXiv  Detail & Related papers  (2021-07-30T04:50:56Z)
• Empirical Analysis of Image Caption Generation using Deep Learning [0.0]
  We have implemented and experimented with various flavors of multi-modal image captioning networks. The goal is to analyze the performance of each approach using various evaluation metrics.
  arXiv  Detail & Related papers  (2021-05-14T05:38:13Z)
• Beyond Single Stage Encoder-Decoder Networks: Deep Decoders for Semantic Image Segmentation [56.44853893149365]
  Single encoder-decoder methodologies for semantic segmentation are reaching their peak in terms of segmentation quality and efficiency per number of layers. We propose a new architecture based on a decoder which uses a set of shallow networks for capturing more information content. In order to further improve the architecture we introduce a weight function which aims to re-balance classes to increase the attention of the networks to under-represented objects.
  arXiv  Detail & Related papers  (2020-07-19T18:44:34Z)
• Rethinking and Improving Natural Language Generation with Layer-Wise Multi-View Decoding [59.48857453699463]
  In sequence-to-sequence learning, the decoder relies on the attention mechanism to efficiently extract information from the encoder. Recent work has proposed to use representations from different encoder layers for diversified levels of information. We propose layer-wise multi-view decoding, where for each decoder layer, together with the representations from the last encoder layer, which serve as a global view, those from other encoder layers are supplemented for a stereoscopic view of the source sequences.
  arXiv  Detail & Related papers  (2020-05-16T20:00:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.