Meta-Learning for Color-to-Infrared Cross-Modal Style Transfer

• URL: http://arxiv.org/abs/2212.12824v1
• Date: Sat, 24 Dec 2022 22:38:16 GMT
• Title: Meta-Learning for Color-to-Infrared Cross-Modal Style Transfer
• Authors: Evelyn A. Stump, Francesco Luzi, Leslie M. Collins, Jordan M. Malof
• Abstract summary: Recent object detection models for infrared (IR) imagery are based upon deep neural networks (DNNs) and require large amounts of labeled training imagery. To address this problem, we explore cross-modal style transfer (CMST) to leverage large and diverse color imagery datasets.
• Score: 5.215058915075776
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Recent object detection models for infrared (IR) imagery are based upon deep neural networks (DNNs) and require large amounts of labeled training imagery. However, publicly-available datasets that can be used for such training are limited in their size and diversity. To address this problem, we explore cross-modal style transfer (CMST) to leverage large and diverse color imagery datasets so that they can be used to train DNN-based IR image based object detectors. We evaluate six contemporary stylization methods on four publicly-available IR datasets - the first comparison of its kind - and find that CMST is highly effective for DNN-based detectors. Surprisingly, we find that existing data-driven methods are outperformed by a simple grayscale stylization (an average of the color channels). Our analysis reveals that existing data-driven methods are either too simplistic or introduce significant artifacts into the imagery. To overcome these limitations, we propose meta-learning style transfer (MLST), which learns a stylization by composing and tuning well-behaved analytic functions. We find that MLST leads to more complex stylizations without introducing significant image artifacts and achieves the best overall detector performance on our benchmark datasets.

Related papers

• Unleashing the Power of Generic Segmentation Models: A Simple Baseline for Infrared Small Target Detection [57.666055329221194]
  We investigate the adaptation of generic segmentation models, such as the Segment Anything Model (SAM), to infrared small object detection tasks. Our model demonstrates significantly improved performance in both accuracy and throughput compared to existing approaches.
  arXiv  Detail & Related papers  (2024-09-07T05:31:24Z)
• Modality Translation for Object Detection Adaptation Without Forgetting Prior Knowledge [11.905387325966311]
  This paper focuses on adapting a large object detection model trained on RGB images to new data extracted from IR images. We propose Modality Translator (ModTr) as an alternative to the common approach of fine-tuning a large model to the new modality.
  arXiv  Detail & Related papers  (2024-04-01T21:28:50Z)
• SIRST-5K: Exploring Massive Negatives Synthesis with Self-supervised Learning for Robust Infrared Small Target Detection [53.19618419772467]
  Single-frame infrared small target (SIRST) detection aims to recognize small targets from clutter backgrounds. With the development of Transformer, the scale of SIRST models is constantly increasing. With a rich diversity of infrared small target data, our algorithm significantly improves the model performance and convergence speed.
  arXiv  Detail & Related papers  (2024-03-08T16:14:54Z)
• Tensor Factorization for Leveraging Cross-Modal Knowledge in Data-Constrained Infrared Object Detection [22.60228799622782]
  Key bottleneck in object detection in IR images is lack of sufficient labeled training data. We seek to leverage cues from the RGB modality to scale object detectors to the IR modality, while preserving model performance in the RGB modality. We first pretrain these factor matrices on the RGB modality, for which plenty of training data are assumed to exist and then augment only a few trainable parameters for training on the IR modality to avoid over-fitting.
  arXiv  Detail & Related papers  (2023-09-28T16:55:52Z)
• Clothes Grasping and Unfolding Based on RGB-D Semantic Segmentation [21.950751953721817]
  We propose a novel Bi-directional Fractal Cross Fusion Network (BiFCNet) for semantic segmentation. We use RGB images with rich color features as input to our network in which the Fractal Cross Fusion module fuses RGB and depth data. To reduce the cost of real data collection, we propose a data augmentation method based on an adversarial strategy.
  arXiv  Detail & Related papers  (2023-05-05T03:21:55Z)
• Multi-Spectral Image Classification with Ultra-Lean Complex-Valued Models [28.798100220715686]
  Multi-spectral imagery is invaluable for remote sensing due to different spectral signatures exhibited by materials. We apply complex-valued co-domain symmetric models to classify real-valued MSI images. Our work is the first to demonstrate the value of complex-valued deep learning on real-valued MSI data.
  arXiv  Detail & Related papers  (2022-11-21T19:01:53Z)
• Auto-Transfer: Learning to Route Transferrable Representations [77.30427535329571]
  We propose a novel adversarial multi-armed bandit approach which automatically learns to route source representations to appropriate target representations. We see upwards of 5% accuracy improvements compared with the state-of-the-art knowledge transfer methods.
  arXiv  Detail & Related papers  (2022-02-02T13:09:27Z)
• Multi-Scale Iterative Refinement Network for RGB-D Salient Object Detection [7.062058947498447]
  salient visual cues appear in various scales and resolutions of RGB images due to semantic gaps at different feature levels. Similar salient patterns are available in cross-modal depth images as well as multi-scale versions. We devise attention based fusion module (ABF) to address on cross-modal correlation.
  arXiv  Detail & Related papers  (2022-01-24T10:33:00Z)
• Infrared Small-Dim Target Detection with Transformer under Complex Backgrounds [155.388487263872]
  We propose a new infrared small-dim target detection method with the transformer. We adopt the self-attention mechanism of the transformer to learn the interaction information of image features in a larger range. We also design a feature enhancement module to learn more features of small-dim targets.
  arXiv  Detail & Related papers  (2021-09-29T12:23:41Z)
• Self-Supervised Representation Learning for RGB-D Salient Object Detection [93.17479956795862]
  We use Self-Supervised Representation Learning to design two pretext tasks: the cross-modal auto-encoder and the depth-contour estimation. Our pretext tasks require only a few and un RGB-D datasets to perform pre-training, which make the network capture rich semantic contexts. For the inherent problem of cross-modal fusion in RGB-D SOD, we propose a multi-path fusion module.
  arXiv  Detail & Related papers  (2021-01-29T09:16:06Z)
• Anchor-free Small-scale Multispectral Pedestrian Detection [88.7497134369344]
  We propose a method for effective and efficient multispectral fusion of the two modalities in an adapted single-stage anchor-free base architecture. We aim at learning pedestrian representations based on object center and scale rather than direct bounding box predictions. Results show our method's effectiveness in detecting small-scaled pedestrians.
  arXiv  Detail & Related papers  (2020-08-19T13:13:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.