Changen2: Multi-Temporal Remote Sensing Generative Change Foundation Model

• URL: http://arxiv.org/abs/2406.17998v1
• Date: Wed, 26 Jun 2024 01:03:39 GMT
• Title: Changen2: Multi-Temporal Remote Sensing Generative Change Foundation Model
• Authors: Zhuo Zheng, Stefano Ermon, Dongjun Kim, Liangpei Zhang, Yanfei Zhong,
• Abstract summary: We present change data generators based on generative models which are cheap and automatic. Changen2 is a generative change foundation model that can be trained at scale via self-supervision. The resulting model possesses inherent zero-shot change detection capabilities and excellent transferability.
• Score: 62.337749660637755
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Our understanding of the temporal dynamics of the Earth's surface has been advanced by deep vision models, which often require lots of labeled multi-temporal images for training. However, collecting, preprocessing, and annotating multi-temporal remote sensing images at scale is non-trivial since it is expensive and knowledge-intensive. In this paper, we present change data generators based on generative models, which are cheap and automatic, alleviating these data problems. Our main idea is to simulate a stochastic change process over time. We describe the stochastic change process as a probabilistic graphical model (GPCM), which factorizes the complex simulation problem into two more tractable sub-problems, i.e., change event simulation and semantic change synthesis. To solve these two problems, we present Changen2, a GPCM with a resolution-scalable diffusion transformer which can generate time series of images and their semantic and change labels from labeled or unlabeled single-temporal images. Changen2 is a generative change foundation model that can be trained at scale via self-supervision, and can produce change supervisory signals from unlabeled single-temporal images. Unlike existing foundation models, Changen2 synthesizes change data to train task-specific foundation models for change detection. The resulting model possesses inherent zero-shot change detection capabilities and excellent transferability. Experiments suggest Changen2 has superior spatiotemporal scalability, e.g., Changen2 model trained on 256$^2$ pixel single-temporal images can yield time series of any length and resolutions of 1,024$^2$ pixels. Changen2 pre-trained models exhibit superior zero-shot performance (narrowing the performance gap to 3% on LEVIR-CD and approximately 10% on both S2Looking and SECOND, compared to fully supervised counterparts) and transferability across multiple types of change tasks.

Related papers

• Integrated Dynamic Phenological Feature for Remote Sensing Image Land Cover Change Detection [5.109855690325439]
  We introduce the InPhea model, which integrates phenological features into a remote sensing image CD framework. A constrainer with four constraint modules and a multi-stage contrastive learning approach is employed to aid in the model's understanding of phenological characteristics. Experiments on the HRSCD, SECD, and PSCD-Wuhan datasets reveal that InPhea outperforms other models.
  arXiv  Detail & Related papers  (2024-08-08T01:07:28Z)
• Counterfactual Image Editing [54.21104691749547]
  Counterfactual image editing is an important task in generative AI, which asks how an image would look if certain features were different. We formalize the counterfactual image editing task using formal language, modeling the causal relationships between latent generative factors and images. We develop an efficient algorithm to generate counterfactual images by leveraging neural causal models.
  arXiv  Detail & Related papers  (2024-02-07T20:55:39Z)
• Scalable Multi-Temporal Remote Sensing Change Data Generation via Simulating Stochastic Change Process [21.622442722863028]
  We present a scalable multi-temporal remote sensing change data generator via generative modeling. Our main idea is to simulate a change process over time. To solve these two problems, we present the change generator (Changen), a GAN-based GPCM, enabling controllable object change data generation.
  arXiv  Detail & Related papers  (2023-09-29T07:37:26Z)
• Self-Pair: Synthesizing Changes from Single Source for Object Change Detection in Remote Sensing Imagery [6.586756080460231]
  We train a change detector using two spatially unrelated images with corresponding semantic labels such as building. We show that manipulating the source image as an after-image is crucial to the performance of change detection. Our method outperforms existing methods based on single-temporal supervision.
  arXiv  Detail & Related papers  (2022-12-20T13:26:42Z)
• TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis [80.56913334060404]
  Time series analysis is of immense importance in applications, such as weather forecasting, anomaly detection, and action recognition. Previous methods attempt to accomplish this directly from the 1D time series. We ravel out the complex temporal variations into the multiple intraperiod- and interperiod-variations.
  arXiv  Detail & Related papers  (2022-10-05T12:19:51Z)
• ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond [76.35955924137986]
  We propose a Vision Transformer Advanced by Exploring intrinsic IB from convolutions, i.e., ViTAE. ViTAE has several spatial pyramid reduction modules to downsample and embed the input image into tokens with rich multi-scale context. We obtain the state-of-the-art classification performance, i.e., 88.5% Top-1 classification accuracy on ImageNet validation set and the best 91.2% Top-1 accuracy on ImageNet real validation set.
  arXiv  Detail & Related papers  (2022-02-21T10:40:05Z)
• Leveraging Image-based Generative Adversarial Networks for Time Series Generation [4.541582055558865]
  We propose a two-dimensional image representation for time series, the Extended Intertemporal Return Plot (XIRP) Our approach captures the intertemporal time series dynamics in a scale-invariant and invertible way, reducing training time and improving sample quality.
  arXiv  Detail & Related papers  (2021-12-15T11:55:11Z)
• DiffuseMorph: Unsupervised Deformable Image Registration Along Continuous Trajectory Using Diffusion Models [31.826844124173984]
  We present a novel approach of diffusion model-based probabilistic image registration, called DiffuseMorph. Our model learns the score function of the deformation between moving and fixed images. Our method can provide flexible and accurate deformation with a capability of topology preservation.
  arXiv  Detail & Related papers  (2021-12-09T08:41:23Z)
• Diverse Image Inpainting with Bidirectional and Autoregressive Transformers [55.21000775547243]
  We propose BAT-Fill, an image inpainting framework with a novel bidirectional autoregressive transformer (BAT) BAT-Fill inherits the merits of transformers and CNNs in a two-stage manner, which allows to generate high-resolution contents without being constrained by the quadratic complexity of attention in transformers.
  arXiv  Detail & Related papers  (2021-04-26T03:52:27Z)
• Encoding Robustness to Image Style via Adversarial Feature Perturbations [72.81911076841408]
  We adapt adversarial training by directly perturbing feature statistics, rather than image pixels, to produce robust models. Our proposed method, Adversarial Batch Normalization (AdvBN), is a single network layer that generates worst-case feature perturbations during training.
  arXiv  Detail & Related papers  (2020-09-18T17:52:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.