Diffusion-RSCC: Diffusion Probabilistic Model for Change Captioning in Remote Sensing Images

• URL: http://arxiv.org/abs/2405.12875v1
• Date: Tue, 21 May 2024 15:44:31 GMT
• Title: Diffusion-RSCC: Diffusion Probabilistic Model for Change Captioning in Remote Sensing Images
• Authors: Xiaofei Yu, Yitong Li, Jie Ma,
• Abstract summary: RSICC aims at generating human-like language to describe semantic changes between bi-temporal remote sensing image pairs. Inspired by the remarkable generative power of diffusion model, we propose a probabilistic diffusion model for RSICC. In training process, we construct a noise predictor conditioned on cross modal features to learn the distribution from the real caption distribution to the standard Gaussian distribution under the Markov chain. In testing phase, the well-trained noise predictor helps to estimate the mean value of the distribution and generate change captions step by step.
• Score: 14.236580915897585
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Remote sensing image change captioning (RSICC) aims at generating human-like language to describe the semantic changes between bi-temporal remote sensing image pairs. It provides valuable insights into environmental dynamics and land management. Unlike conventional change captioning task, RSICC involves not only retrieving relevant information across different modalities and generating fluent captions, but also mitigating the impact of pixel-level differences on terrain change localization. The pixel problem due to long time span decreases the accuracy of generated caption. Inspired by the remarkable generative power of diffusion model, we propose a probabilistic diffusion model for RSICC to solve the aforementioned problems. In training process, we construct a noise predictor conditioned on cross modal features to learn the distribution from the real caption distribution to the standard Gaussian distribution under the Markov chain. Meanwhile, a cross-mode fusion and a stacking self-attention module are designed for noise predictor in the reverse process. In testing phase, the well-trained noise predictor helps to estimate the mean value of the distribution and generate change captions step by step. Extensive experiments on the LEVIR-CC dataset demonstrate the effectiveness of our Diffusion-RSCC and its individual components. The quantitative results showcase superior performance over existing methods across both traditional and newly augmented metrics. The code and materials will be available online at https://github.com/Fay-Y/Diffusion-RSCC.

Related papers

• Diffusion-Aided Joint Source Channel Coding For High Realism Wireless Image Transmission [24.372996233209854]
  DiffJSCC is a novel framework that produces high-realism images via the conditional diffusion denoising process. It can achieve highly realistic reconstructions for 768x512 pixel Kodak images with only 3072 symbols.
  arXiv  Detail & Related papers  (2024-04-27T00:12:13Z)
• Generalized Consistency Trajectory Models for Image Manipulation [59.576781858809355]
  Diffusion models (DMs) excel in unconditional generation, as well as on applications such as image editing and restoration. This work aims to unlock the full potential of consistency trajectory models (CTMs) by proposing generalized CTMs (GCTMs) We discuss the design space of GCTMs and demonstrate their efficacy in various image manipulation tasks such as image-to-image translation, restoration, and editing.
  arXiv  Detail & Related papers  (2024-03-19T07:24:54Z)
• Adaptive Semantic-Enhanced Denoising Diffusion Probabilistic Model for Remote Sensing Image Super-Resolution [7.252121550658619]
  Denoising Diffusion Probabilistic Model (DDPM) has shown promising performance in image reconstructions. High-frequency details generated by DDPM often suffer from misalignment with HR images due to model's tendency to overlook long-range semantic contexts. An adaptive semantic-enhanced DDPM (ASDDPM) is proposed to enhance the detail-preserving capability of the DDPM.
  arXiv  Detail & Related papers  (2024-03-17T04:08:58Z)
• Improving Diffusion-Based Image Synthesis with Context Prediction [49.186366441954846]
  Existing diffusion models mainly try to reconstruct input image from a corrupted one with a pixel-wise or feature-wise constraint along spatial axes. We propose ConPreDiff to improve diffusion-based image synthesis with context prediction. Our ConPreDiff consistently outperforms previous methods and achieves a new SOTA text-to-image generation results on MS-COCO, with a zero-shot FID score of 6.21.
  arXiv  Detail & Related papers  (2024-01-04T01:10:56Z)
• Affine-Transformation-Invariant Image Classification by Differentiable Arithmetic Distribution Module [8.125023712173686]
  Convolutional Neural Networks (CNNs) have achieved promising results in image classification. CNNs are vulnerable to affine transformations including rotation, translation, flip and shuffle. In this work, we introduce a more robust substitute by incorporating distribution learning techniques.
  arXiv  Detail & Related papers  (2023-09-01T22:31:32Z)
• Diffusion in Diffusion: Cyclic One-Way Diffusion for Text-Vision-Conditioned Generation [11.80682025950519]
  In this work, we investigate the diffusion (physics) in diffusion (machine learning) properties. We propose our Cyclic One-Way Diffusion (COW) method to control the direction of diffusion phenomenon. Our method provides a novel perspective to understand the task needs and is applicable to a wider range of customization scenarios.
  arXiv  Detail & Related papers  (2023-06-14T05:25:06Z)
• Semantic-Conditional Diffusion Networks for Image Captioning [116.86677915812508]
  We propose a new diffusion model based paradigm tailored for image captioning, namely Semantic-Conditional Diffusion Networks (SCD-Net) In SCD-Net, multiple Diffusion Transformer structures are stacked to progressively strengthen the output sentence with better visional-language alignment and linguistical coherence. Experiments on COCO dataset demonstrate the promising potential of using diffusion models in the challenging image captioning task.
  arXiv  Detail & Related papers  (2022-12-06T16:08:16Z)
• Diffusion Visual Counterfactual Explanations [51.077318228247925]
  Visual Counterfactual Explanations (VCEs) are an important tool to understand the decisions of an image. Current approaches for the generation of VCEs are restricted to adversarially robust models and often contain non-realistic artefacts. In this paper, we overcome this by generating Visual Diffusion Counterfactual Explanations (DVCEs) for arbitrary ImageNet classifiers.
  arXiv  Detail & Related papers  (2022-10-21T09:35:47Z)
• f-DM: A Multi-stage Diffusion Model via Progressive Signal Transformation [56.04628143914542]
  Diffusion models (DMs) have recently emerged as SoTA tools for generative modeling in various domains. We propose f-DM, a generalized family of DMs which allows progressive signal transformation. We apply f-DM in image generation tasks with a range of functions, including down-sampling, blurring, and learned transformations.
  arXiv  Detail & Related papers  (2022-10-10T18:49:25Z)
• Treatment Learning Causal Transformer for Noisy Image Classification [62.639851972495094]
  In this work, we incorporate this binary information of "existence of noise" as treatment into image classification tasks to improve prediction accuracy. Motivated from causal variational inference, we propose a transformer-based architecture, that uses a latent generative model to estimate robust feature representations for noise image classification. We also create new noisy image datasets incorporating a wide range of noise factors for performance benchmarking.
  arXiv  Detail & Related papers  (2022-03-29T13:07:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.