On the Influence of Shape, Texture and Color for Learning Semantic Segmentation

• URL: http://arxiv.org/abs/2410.14878v1
• Date: Fri, 18 Oct 2024 21:52:02 GMT
• Title: On the Influence of Shape, Texture and Color for Learning Semantic Segmentation
• Abstract summary: In recent years, a body of works has emerged, studying shape and texture biases of off-the-shelf pre-trained deep neural networks (DNN) for image classification. We study these questions on semantic segmentation which allows us to address our questions on pixel level. Our study on three datasets reveals that neither texture nor shape clearly dominate the learning success, however a combination of shape and color but without texture achieves surprisingly strong results.
• Score: 5.172964916120902
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In recent years, a body of works has emerged, studying shape and texture biases of off-the-shelf pre-trained deep neural networks (DNN) for image classification. These works study how much a trained DNN relies on image cues, predominantly shape and texture. In this work, we switch the perspective, posing the following questions: What can a DNN learn from each of the image cues, i.e., shape, texture and color, respectively? How much does each cue influence the learning success? And what are the synergy effects between different cues? Studying these questions sheds light upon cue influences on learning and thus the learning capabilities of DNNs. We study these questions on semantic segmentation which allows us to address our questions on pixel level. To conduct this study, we develop a generic procedure to decompose a given dataset into multiple ones, each of them only containing either a single cue or a chosen mixture. This framework is then applied to two real-world datasets, Cityscapes and PASCAL Context, and a synthetic data set based on the CARLA simulator. We learn the given semantic segmentation task from these cue datasets, creating cue experts. Early fusion of cues is performed by constructing appropriate datasets. This is complemented by a late fusion of experts which allows us to study cue influence location-dependent on pixel level. Our study on three datasets reveals that neither texture nor shape clearly dominate the learning success, however a combination of shape and color but without texture achieves surprisingly strong results. Our findings hold for convolutional and transformer backbones. In particular, qualitatively there is almost no difference in how both of the architecture types extract information from the different cues.

Related papers

• Effect of Rotation Angle in Self-Supervised Pre-training is Dataset-Dependent [3.434553688053531]
  Self-supervised learning for pre-training can help the network learn better low-level features. In contrastive pre-training, the network is pre-trained to distinguish between different versions of the input. We show that, when training using contrastive pre-training in this way, the angle $theta$ and the dataset interact in interesting ways.
  arXiv  Detail & Related papers  (2024-06-21T12:25:07Z)
• Self-supervised Learning of LiDAR 3D Point Clouds via 2D-3D Neural Calibration [107.61458720202984]
  This paper introduces a novel self-supervised learning framework for enhancing 3D perception in autonomous driving scenes. We propose the learnable transformation alignment to bridge the domain gap between image and point cloud data. We establish dense 2D-3D correspondences to estimate the rigid pose.
  arXiv  Detail & Related papers  (2024-01-23T02:41:06Z)
• Unveiling Backbone Effects in CLIP: Exploring Representational Synergies and Variances [49.631908848868505]
  Contrastive Language-Image Pretraining (CLIP) stands out as a prominent method for image representation learning. We investigate the differences in CLIP performance among various neural architectures. We propose a simple, yet effective approach to combine predictions from multiple backbones, leading to a notable performance boost of up to 6.34%.
  arXiv  Detail & Related papers  (2023-12-22T03:01:41Z)
• On the ability of CNNs to extract color invariant intensity based features for image classification [4.297070083645049]
  Convolutional neural networks (CNNs) have demonstrated remarkable success in vision-related tasks. Recent studies suggest that CNNs exhibit a bias toward texture instead of object shape in image classification tasks. This paper investigates the ability of CNNs to adapt to different color distributions in an image while maintaining context and background.
  arXiv  Detail & Related papers  (2023-07-13T00:36:55Z)
• Impact of Colour Variation on Robustness of Deep Neural Networks [0.0]
  Deep neural networks (DNNs) have shown state-of-the-art performance for computer vision applications like image classification, segmentation and object detection. Recent advances have shown their vulnerability to manual digital perturbations in the input data, namely adversarial attacks. In this work, we propose a color-variation dataset by distorting their RGB color on a subset of the ImageNet with 27 different combinations.
  arXiv  Detail & Related papers  (2022-09-02T08:16:04Z)
• Contrastive Learning of Features between Images and LiDAR [18.211513930388417]
  This work treats learning cross-modal features as a dense contrastive learning problem. To learn good features and not lose generality, we developed a variant of widely used PointNet++ architecture for images. We show that our models indeed learn information from both images as well as LiDAR by visualizing the features.
  arXiv  Detail & Related papers  (2022-06-24T04:35:23Z)
• Investigating Neural Architectures by Synthetic Dataset Design [14.317837518705302]
  Recent years have seen the emergence of many new neural network structures (architectures and layers) We sketch a methodology to measure the effect of each structure on a network's ability, by designing ad hoc synthetic datasets. We illustrate our methodology by building three datasets to evaluate each of the three following network properties.
  arXiv  Detail & Related papers  (2022-04-23T10:50:52Z)
• Learning Co-segmentation by Segment Swapping for Retrieval and Discovery [67.6609943904996]
  The goal of this work is to efficiently identify visually similar patterns from a pair of images. We generate synthetic training pairs by selecting object segments in an image and copy-pasting them into another image. We show our approach provides clear improvements for artwork details retrieval on the Brueghel dataset.
  arXiv  Detail & Related papers  (2021-10-29T16:51:16Z)
• Shape or Texture: Understanding Discriminative Features in CNNs [28.513300496205044]
  Recent studies have shown that CNNs actually exhibit a texture bias' We show that a network learns the majority of overall shape information at the first few epochs of training. We also show that the encoding of shape does not imply the encoding of localized per-pixel semantic information.
  arXiv  Detail & Related papers  (2021-01-27T18:54:00Z)
• Assessing The Importance Of Colours For CNNs In Object Recognition [70.70151719764021]
  Convolutional neural networks (CNNs) have been shown to exhibit conflicting properties. We demonstrate that CNNs often rely heavily on colour information while making a prediction. We evaluate a model trained with congruent images on congruent, greyscale, and incongruent images.
  arXiv  Detail & Related papers  (2020-12-12T22:55:06Z)
• Leveraging Tacit Information Embedded in CNN Layers for Visual Tracking [1.7188280334580193]
  We propose an adaptive combination of several CNN layers in a single DCF tracker to address variations of the target appearances. Experiments demonstrate that using the additional implicit data layers of CNNs significantly improves the tracker.
  arXiv  Detail & Related papers  (2020-10-02T21:16:26Z)
• Informative Dropout for Robust Representation Learning: A Shape-bias Perspective [84.30946377024297]
  We propose a light-weight model-agnostic method, namely Informative Dropout (InfoDrop), to improve interpretability and reduce texture bias. Specifically, we discriminate texture from shape based on local self-information in an image, and adopt a Dropout-like algorithm to decorrelate the model output from the local texture.
  arXiv  Detail & Related papers  (2020-08-10T16:52:24Z)
• Teaching CNNs to mimic Human Visual Cognitive Process & regularise Texture-Shape bias [18.003188982585737]
  Recent experiments in computer vision demonstrate texture bias as the primary reason for supreme results in models employing Convolutional Neural Networks (CNNs) It is believed that the cost function forces the CNN to take a greedy approach and develop a proclivity for local information like texture to increase accuracy, thus failing to explore any global statistics. We propose CognitiveCNN, a new intuitive architecture, inspired from feature integration theory in psychology to utilise human interpretable feature like shape, texture, edges etc. to reconstruct, and classify the image.
  arXiv  Detail & Related papers  (2020-06-25T22:32:54Z)
• What Do Neural Networks Learn When Trained With Random Labels? [20.54410239839646]
  We study deep neural networks (DNNs) trained on natural image data with entirely random labels. We show analytically for convolutional and fully connected networks that an alignment between the principal components of network parameters and data takes place when training with random labels. We show how this alignment produces a positive transfer: networks pre-trained with random labels train faster downstream compared to training from scratch.
  arXiv  Detail & Related papers  (2020-06-18T12:07:22Z)
• Self-supervised Learning on Graphs: Deep Insights and New Direction [66.78374374440467]
  Self-supervised learning (SSL) aims to create domain specific pretext tasks on unlabeled data. There are increasing interests in generalizing deep learning to the graph domain in the form of graph neural networks (GNNs)
  arXiv  Detail & Related papers  (2020-06-17T20:30:04Z)
• SPIN: Structure-Preserving Inner Offset Network for Scene Text Recognition [48.676064155070556]
  Arbitrary text appearance poses a great challenge in scene text recognition tasks. We introduce a new learnable geometric-unrelated module, the Structure-Preserving Inner Offset Network (SPIN) SPIN allows the color manipulation of source data within the network.
  arXiv  Detail & Related papers  (2020-05-27T01:47:07Z)
• Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision [57.14468881854616]
  We propose an auxiliary training objective that improves the generalization capabilities of neural networks. We use pairs of minimally-different examples with different labels, a.k.a counterfactual or contrasting examples, which provide a signal indicative of the underlying causal structure of the task. Models trained with this technique demonstrate improved performance on out-of-distribution test sets.
  arXiv  Detail & Related papers  (2020-04-20T02:47:49Z)
• Linguistically Driven Graph Capsule Network for Visual Question Reasoning [153.76012414126643]
  We propose a hierarchical compositional reasoning model called the "Linguistically driven Graph Capsule Network" The compositional process is guided by the linguistic parse tree. Specifically, we bind each capsule in the lowest layer to bridge the linguistic embedding of a single word in the original question with visual evidence. Experiments on the CLEVR dataset, CLEVR compositional generation test, and FigureQA dataset demonstrate the effectiveness and composition generalization ability of our end-to-end model.
  arXiv  Detail & Related papers  (2020-03-23T03:34:25Z)
• Hold me tight! Influence of discriminative features on deep network boundaries [63.627760598441796]
  We propose a new perspective that relates dataset features to the distance of samples to the decision boundary. This enables us to carefully tweak the position of the training samples and measure the induced changes on the boundaries of CNNs trained on large-scale vision datasets.
  arXiv  Detail & Related papers  (2020-02-15T09:29:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.