Non-Visible Light Data Synthesis and Application: A Case Study for
Synthetic Aperture Radar Imagery
- URL: http://arxiv.org/abs/2311.17486v1
- Date: Wed, 29 Nov 2023 09:48:01 GMT
- Title: Non-Visible Light Data Synthesis and Application: A Case Study for
Synthetic Aperture Radar Imagery
- Authors: Zichen Tian, Zhaozheng Chen, Qianru Sun
- Abstract summary: We explore the "hidden" ability of large-scale pre-trained image generation models, such as Stable Diffusion and Imagen, in non-visible light domains.
We propose a 2-stage low-rank adaptation method, and we call it 2LoRA.
In the first stage, the model is adapted using aerial-view regular image data (whose structure matches SAR), followed by the second stage where the base model from the first stage is further adapted using SAR modality data.
- Score: 30.590315753622132
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: We explore the "hidden" ability of large-scale pre-trained image generation
models, such as Stable Diffusion and Imagen, in non-visible light domains,
taking Synthetic Aperture Radar (SAR) data for a case study. Due to the
inherent challenges in capturing satellite data, acquiring ample SAR training
samples is infeasible. For instance, for a particular category of ship in the
open sea, we can collect only few-shot SAR images which are too limited to
derive effective ship recognition models. If large-scale models pre-trained
with regular images can be adapted to generating novel SAR images, the problem
is solved. In preliminary study, we found that fine-tuning these models with
few-shot SAR images is not working, as the models can not capture the two
primary differences between SAR and regular images: structure and modality. To
address this, we propose a 2-stage low-rank adaptation method, and we call it
2LoRA. In the first stage, the model is adapted using aerial-view regular image
data (whose structure matches SAR), followed by the second stage where the base
model from the first stage is further adapted using SAR modality data.
Particularly in the second stage, we introduce a novel prototype LoRA (pLoRA),
as an improved version of 2LoRA, to resolve the class imbalance problem in SAR
datasets. For evaluation, we employ the resulting generation model to
synthesize additional SAR data. This augmentation, when integrated into the
training process of SAR classification as well as segmentation models, yields
notably improved performance for minor classes
Related papers
- SARATR-X: A Foundation Model for Synthetic Aperture Radar Images Target Recognition [19.776275680586977]
This paper aims to achieve general SAR ATR based on a foundation model with Self-Supervised Learning (SSL)
A foundation model named SARATR-X is proposed with the following four aspects.
First, we integrated 14 datasets with various target categories and imaging conditions as a pre-training dataset. Second, different model backbones were discussed to find the most suitable approaches for remote-sensing images. Third, we applied two-stage training and SAR gradient features to ensure the diversity and scalability of SARATR-X.
arXiv Detail & Related papers (2024-05-15T14:17:44Z) - SatSynth: Augmenting Image-Mask Pairs through Diffusion Models for Aerial Semantic Segmentation [69.42764583465508]
We explore the potential of generative image diffusion to address the scarcity of annotated data in earth observation tasks.
To the best of our knowledge, we are the first to generate both images and corresponding masks for satellite segmentation.
arXiv Detail & Related papers (2024-03-25T10:30:22Z) - Predicting Gradient is Better: Exploring Self-Supervised Learning for SAR ATR with a Joint-Embedding Predictive Architecture [23.375515181854254]
This study investigates an effective Self-Supervised Learning (SSL) method for SAR Automatic Target Recognition (ATR)
SSL aims to construct supervision signals directly from the data, which minimizes the need for expensive expert annotation.
We present a novel Joint-Embedding Predictive Architecture for SAR ATR (SAR-JEPA), which leverages local masked patches to predict the multi-scale SAR gradient representations of unseen context.
arXiv Detail & Related papers (2023-11-26T01:05:55Z) - Efficient Test-Time Adaptation for Super-Resolution with Second-Order
Degradation and Reconstruction [62.955327005837475]
Image super-resolution (SR) aims to learn a mapping from low-resolution (LR) to high-resolution (HR) using paired HR-LR training images.
We present an efficient test-time adaptation framework for SR, named SRTTA, which is able to quickly adapt SR models to test domains with different/unknown degradation types.
arXiv Detail & Related papers (2023-10-29T13:58:57Z) - Towards Lightweight Super-Resolution with Dual Regression Learning [58.98801753555746]
Deep neural networks have exhibited remarkable performance in image super-resolution (SR) tasks.
The SR problem is typically an ill-posed problem and existing methods would come with several limitations.
We propose a dual regression learning scheme to reduce the space of possible SR mappings.
arXiv Detail & Related papers (2022-07-16T12:46:10Z) - More Photos are All You Need: Semi-Supervised Learning for Fine-Grained
Sketch Based Image Retrieval [112.1756171062067]
We introduce a novel semi-supervised framework for cross-modal retrieval.
At the centre of our design is a sequential photo-to-sketch generation model.
We also introduce a discriminator guided mechanism to guide against unfaithful generation.
arXiv Detail & Related papers (2021-03-25T17:27:08Z) - Sparse Signal Models for Data Augmentation in Deep Learning ATR [0.8999056386710496]
We propose a data augmentation approach to incorporate domain knowledge and improve the generalization power of a data-intensive learning algorithm.
We exploit the sparsity of the scattering centers in the spatial domain and the smoothly-varying structure of the scattering coefficients in the azimuthal domain to solve the ill-posed problem of over-parametrized model fitting.
arXiv Detail & Related papers (2020-12-16T21:46:33Z) - Speckle2Void: Deep Self-Supervised SAR Despeckling with Blind-Spot
Convolutional Neural Networks [30.410981386006394]
despeckling is a crucial preliminary step in scene analysis algorithms.
Recent success of deep learning envisions a new generation of despeckling techniques.
We propose a self-supervised Bayesian despeckling method.
arXiv Detail & Related papers (2020-07-04T11:38:48Z) - SAR2SAR: a semi-supervised despeckling algorithm for SAR images [3.9490074068698]
Deep learning algorithm with self-supervision is proposed in this paper: SAR2SAR.
The strategy to adapt it to SAR despeckling is presented, based on a compensation of temporal changes and a loss function adapted to the statistics of speckle.
Results on real images are discussed, to show the potential of the proposed algorithm.
arXiv Detail & Related papers (2020-06-26T15:07:28Z) - Unlimited Resolution Image Generation with R2D2-GANs [69.90258455164513]
We present a novel simulation technique for generating high quality images of any predefined resolution.
This method can be used to synthesize sonar scans of size equivalent to those collected during a full-length mission.
The data produced is continuous, realistically-looking, and can also be generated at least two times faster than the real speed of acquisition.
arXiv Detail & Related papers (2020-03-02T17:49:32Z) - Characteristic Regularisation for Super-Resolving Face Images [81.84939112201377]
Existing facial image super-resolution (SR) methods focus mostly on improving artificially down-sampled low-resolution (LR) imagery.
Previous unsupervised domain adaptation (UDA) methods address this issue by training a model using unpaired genuine LR and HR data.
This renders the model overstretched with two tasks: consistifying the visual characteristics and enhancing the image resolution.
We formulate a method that joins the advantages of conventional SR and UDA models.
arXiv Detail & Related papers (2019-12-30T16:27:24Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.