Planckian jitter: enhancing the color quality of self-supervised visual representations

• URL: http://arxiv.org/abs/2202.07993v1
• Date: Wed, 16 Feb 2022 11:13:37 GMT
• Title: Planckian jitter: enhancing the color quality of self-supervised visual representations
• Authors: Simone Zini, Marco Buzzelli, Bart{\l}omiej Twardowski and Joost van de Weijer
• Abstract summary: We analyze how the traditionally used color jitter negatively impacts the quality of the color features in the learned feature representation. We replace this module with physics-based color augmentation, called Planckian jitter, which creates realistic variations in chromaticity. We show that the performance of the learned features is robust with respect to illuminant variations.
• Score: 32.28858433165915
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Several recent works on self-supervised learning are trained by mapping different augmentations of the same image to the same feature representation. The set of used data augmentations is of crucial importance for the quality of the learned feature representation. We analyze how the traditionally used color jitter negatively impacts the quality of the color features in the learned feature representation. To address this problem, we replace this module with physics-based color augmentation, called Planckian jitter, which creates realistic variations in chromaticity, producing a model robust to llumination changes that can be commonly observed in real life, while maintaining the ability to discriminate the image content based on color information. We further improve the performance by introducing a latent space combination of color-sensitive and non-color-sensitive features. These are found to be complementary and the combination leads to large absolute performance gains over the default data augmentation on color classification tasks, including on Flowers-102 (+15%), Cub200 (+11%), VegFru (+15%), and T1K+ (+12%). Finally, we present a color sensitivity analysis to document the impact of different training methods on the model neurons and we show that the performance of the learned features is robust with respect to illuminant variations.

Related papers

• Data Augmentation via Latent Diffusion for Saliency Prediction [67.88936624546076]
  Saliency prediction models are constrained by the limited diversity and quantity of labeled data. We propose a novel data augmentation method for deep saliency prediction that edits natural images while preserving the complexity and variability of real-world scenes.
  arXiv  Detail & Related papers  (2024-09-11T14:36:24Z)
• Exploring Color Invariance through Image-Level Ensemble Learning [7.254270666779331]
  This study introduces a learning strategy named Random Color Erasing. It selectively erases partial or complete color information in the training data without disrupting the original image structure. This approach mitigates the risk of overfitting and enhances the model's ability to handle color variation.
  arXiv  Detail & Related papers  (2024-01-19T06:04:48Z)
• Diffusing Colors: Image Colorization with Text Guided Diffusion [11.727899027933466]
  We present a novel image colorization framework that utilizes image diffusion techniques with granular text prompts. Our method provides a balance between automation and control, outperforming existing techniques in terms of visual quality and semantic coherence. Our approach holds potential particularly for color enhancement and historical image colorization.
  arXiv  Detail & Related papers  (2023-12-07T08:59:20Z)
• Improving Fairness using Vision-Language Driven Image Augmentation [60.428157003498995]
  Fairness is crucial when training a deep-learning discriminative model, especially in the facial domain. Models tend to correlate specific characteristics (such as age and skin color) with unrelated attributes (downstream tasks) This paper proposes a method to mitigate these correlations to improve fairness.
  arXiv  Detail & Related papers  (2023-11-02T19:51:10Z)
• Color Equivariant Convolutional Networks [50.655443383582124]
  CNNs struggle if there is data imbalance between color variations introduced by accidental recording conditions. We propose Color Equivariant Convolutions ( CEConvs), a novel deep learning building block that enables shape feature sharing across the color spectrum. We demonstrate the benefits of CEConvs in terms of downstream performance to various tasks and improved robustness to color changes, including train-test distribution shifts.
  arXiv  Detail & Related papers  (2023-10-30T09:18:49Z)
• The Influences of Color and Shape Features in Visual Contrastive Learning [0.0]
  This paper investigates the influences of individual image features (e.g., color and shape) to model performance remain ambiguous. Experimental results show that compared with supervised representations, contrastive representations tend to cluster with objects of similar color.
  arXiv  Detail & Related papers  (2023-01-29T15:10:14Z)
• ColorSense: A Study on Color Vision in Machine Visual Recognition [57.916512479603064]
  We collect 110,000 non-trivial human annotations of foreground and background color labels from visual recognition benchmarks. We validate the use of our datasets by demonstrating that the level of color discrimination has a dominating effect on the performance of machine perception models. Our findings suggest that object recognition tasks such as classification and localization are susceptible to color vision bias.
  arXiv  Detail & Related papers  (2022-12-16T18:51:41Z)
• Name Your Colour For the Task: Artificially Discover Colour Naming via Colour Quantisation Transformer [62.75343115345667]
  We propose a novel colour quantisation transformer, CQFormer, that quantises colour space while maintaining machine recognition on the quantised images. We observe the consistent evolution pattern between our artificial colour system and basic colour terms across human languages. Our colour quantisation method also offers an efficient quantisation method that effectively compresses the image storage.
  arXiv  Detail & Related papers  (2022-12-07T03:39:18Z)
• Neural Color Operators for Sequential Image Retouching [62.99812889713773]
  We propose a novel image retouching method by modeling the retouching process as performing a sequence of newly introduced trainable neural color operators. The neural color operator mimics the behavior of traditional color operators and learns pixelwise color transformation while its strength is controlled by a scalar. Our method consistently achieves the best results compared with SOTA methods in both quantitative measures and visual qualities.
  arXiv  Detail & Related papers  (2022-07-17T05:33:19Z)
• Color Invariant Skin Segmentation [17.501659517108884]
  This paper addresses the problem of automatically detecting human skin in images without reliance on color information. A primary motivation of the work has been to achieve results that are consistent across the full range of skin tones. We present a new approach that performs well in the absence of such information.
  arXiv  Detail & Related papers  (2022-04-21T05:07:21Z)
• Structure-Preserving Multi-Domain Stain Color Augmentation using Style-Transfer with Disentangled Representations [0.9051352746190446]
  HistAuGAN can simulate a wide variety of realistic histology stain colors, thus making neural networks stain-invariant when applied during training. Based on a generative adversarial network (GAN) for image-to-image translation, our model disentangles the content of the image, i.e., the morphological tissue structure, from the stain color attributes. It can be trained on multiple domains and, therefore, learns to cover different stain colors as well as other domain-specific variations introduced in the slide preparation and imaging process.
  arXiv  Detail & Related papers  (2021-07-26T17:52:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.