Emergence of Shape Bias in Convolutional Neural Networks through Activation Sparsity

• URL: http://arxiv.org/abs/2310.18894v1
• Date: Sun, 29 Oct 2023 04:07:52 GMT
• Title: Emergence of Shape Bias in Convolutional Neural Networks through Activation Sparsity
• Authors: Tianqin Li, Ziqi Wen, Yangfan Li, Tai Sing Lee
• Abstract summary: Current deep-learning models for object recognition are heavily biased toward texture. In contrast, human visual systems are known to be biased toward shape and structure. We show that sparse coding, a ubiquitous principle in the brain, can in itself introduce shape bias into the network.
• Score: 8.54598311798543
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current deep-learning models for object recognition are known to be heavily biased toward texture. In contrast, human visual systems are known to be biased toward shape and structure. What could be the design principles in human visual systems that led to this difference? How could we introduce more shape bias into the deep learning models? In this paper, we report that sparse coding, a ubiquitous principle in the brain, can in itself introduce shape bias into the network. We found that enforcing the sparse coding constraint using a non-differential Top-K operation can lead to the emergence of structural encoding in neurons in convolutional neural networks, resulting in a smooth decomposition of objects into parts and subparts and endowing the networks with shape bias. We demonstrated this emergence of shape bias and its functional benefits for different network structures with various datasets. For object recognition convolutional neural networks, the shape bias leads to greater robustness against style and pattern change distraction. For the image synthesis generative adversary networks, the emerged shape bias leads to more coherent and decomposable structures in the synthesized images. Ablation studies suggest that sparse codes tend to encode structures, whereas the more distributed codes tend to favor texture. Our code is host at the github repository: \url{https://github.com/Crazy-Jack/nips2023_shape_vs_texture}

Related papers

• Coding schemes in neural networks learning classification tasks [52.22978725954347]
  We investigate fully-connected, wide neural networks learning classification tasks. We show that the networks acquire strong, data-dependent features. Surprisingly, the nature of the internal representations depends crucially on the neuronal nonlinearity.
  arXiv  Detail & Related papers  (2024-06-24T14:50:05Z)
• Graph Neural Networks for Learning Equivariant Representations of Neural Networks [55.04145324152541]
  We propose to represent neural networks as computational graphs of parameters. Our approach enables a single model to encode neural computational graphs with diverse architectures. We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations.
  arXiv  Detail & Related papers  (2024-03-18T18:01:01Z)
• Connecting metrics for shape-texture knowledge in computer vision [1.7785095623975342]
  Deep neural networks remain brittle and susceptible to many changes in the image that do not cause humans to misclassify images. Part of this different behavior may be explained by the type of features humans and deep neural networks use in vision tasks.
  arXiv  Detail & Related papers  (2023-01-25T14:37:42Z)
• Scalar Invariant Networks with Zero Bias [3.428731916567677]
  We show that zero-bias neural networks can perform comparably to biased networks for practical image classification tasks. We prove that zero-bias neural networks are fair in predicting the zero image. The robustness and fairness advantages of zero-bias neural networks may also indicate a promising path towards trustworthy and ethical AI.
  arXiv  Detail & Related papers  (2022-11-15T20:26:07Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Dynamic Inference with Neural Interpreters [72.90231306252007]
  We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules. inputs to the model are routed through a sequence of functions in a way that is end-to-end learned. We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
  arXiv  Detail & Related papers  (2021-10-12T23:22:45Z)
• Predictive coding feedback results in perceived illusory contours in a recurrent neural network [0.0]
  We equip a deep feedforward convolutional network with brain-inspired recurrent dynamics. We show that the perception of illusory contours could involve feedback connections.
  arXiv  Detail & Related papers  (2021-02-03T09:07:09Z)
• Shape or Texture: Understanding Discriminative Features in CNNs [28.513300496205044]
  Recent studies have shown that CNNs actually exhibit a texture bias' We show that a network learns the majority of overall shape information at the first few epochs of training. We also show that the encoding of shape does not imply the encoding of localized per-pixel semantic information.
  arXiv  Detail & Related papers  (2021-01-27T18:54:00Z)
• Informative Dropout for Robust Representation Learning: A Shape-bias Perspective [84.30946377024297]
  We propose a light-weight model-agnostic method, namely Informative Dropout (InfoDrop), to improve interpretability and reduce texture bias. Specifically, we discriminate texture from shape based on local self-information in an image, and adopt a Dropout-like algorithm to decorrelate the model output from the local texture.
  arXiv  Detail & Related papers  (2020-08-10T16:52:24Z)
• Towards Understanding Hierarchical Learning: Benefits of Neural Representations [160.33479656108926]
  In this work, we demonstrate that intermediate neural representations add more flexibility to neural networks. We show that neural representation can achieve improved sample complexities compared with the raw input. Our results characterize when neural representations are beneficial, and may provide a new perspective on why depth is important in deep learning.
  arXiv  Detail & Related papers  (2020-06-24T02:44:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.