What Do Deep Nets Learn? Class-wise Patterns Revealed in the Input Space

• URL: http://arxiv.org/abs/2101.06898v2
• Date: Sat, 6 Feb 2021 05:09:40 GMT
• Title: What Do Deep Nets Learn? Class-wise Patterns Revealed in the Input Space
• Authors: Shihao Zhao, Xingjun Ma, Yisen Wang, James Bailey, Bo Li, Yu-Gang Jiang
• Abstract summary: We propose a method to visualize and understand the class-wise knowledge learned by deep neural networks (DNNs) under different settings. Our method searches for a single predictive pattern in the pixel space to represent the knowledge learned by the model for each class. In the adversarial setting, we show that adversarially trained models tend to learn more simplified shape patterns.
• Score: 88.37185513453758
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural networks (DNNs) are increasingly deployed in different applications to achieve state-of-the-art performance. However, they are often applied as a black box with limited understanding of what knowledge the model has learned from the data. In this paper, we focus on image classification and propose a method to visualize and understand the class-wise knowledge (patterns) learned by DNNs under three different settings including natural, backdoor and adversarial. Different to existing visualization methods, our method searches for a single predictive pattern in the pixel space to represent the knowledge learned by the model for each class. Based on the proposed method, we show that DNNs trained on natural (clean) data learn abstract shapes along with some texture, and backdoored models learn a suspicious pattern for the backdoored class. Interestingly, the phenomenon that DNNs can learn a single predictive pattern for each class indicates that DNNs can learn a backdoor even from clean data, and the pattern itself is a backdoor trigger. In the adversarial setting, we show that adversarially trained models tend to learn more simplified shape patterns. Our method can serve as a useful tool to better understand the knowledge learned by DNNs on different datasets under different settings.

Related papers

• Towards Deep Network Steganography: From Networks to Networks [23.853644434004135]
  We propose deep network steganography for the covert communication of DNN models. Our scheme is learning task oriented, where the learning task of the secret DNN model is disguised into another ordinary learning task. We conduct experiments on both the intra-task steganography and inter-task steganography.
  arXiv  Detail & Related papers  (2023-07-07T08:02:01Z)
• Visual Recognition with Deep Nearest Centroids [57.35144702563746]
  We devise deep nearest centroids (DNC), a conceptually elegant yet surprisingly effective network for large-scale visual recognition. Compared with parametric counterparts, DNC performs better on image classification (CIFAR-10, ImageNet) and greatly boots pixel recognition (ADE20K, Cityscapes)
  arXiv  Detail & Related papers  (2022-09-15T15:47:31Z)
• Visualizing the Emergence of Intermediate Visual Patterns in DNNs [19.043540343193946]
  This paper proposes a method to visualize the discrimination power of intermediate-layer visual patterns encoded by a DNN. We visualize how the DNN gradually learns regional visual patterns in each intermediate layer during the training process. This method also provides new insights into signal-processing behaviors of existing deep-learning techniques, such as adversarial attacks and knowledge distillation.
  arXiv  Detail & Related papers  (2021-11-05T13:49:39Z)
• Reasoning-Modulated Representations [85.08205744191078]
  We study a common setting where our task is not purely opaque. Our approach paves the way for a new class of data-efficient representation learning.
  arXiv  Detail & Related papers  (2021-07-19T13:57:13Z)
• Wider Vision: Enriching Convolutional Neural Networks via Alignment to External Knowledge Bases [0.3867363075280543]
  We aim to explain and expand CNNs models via the mirroring or alignment of CNN to an external knowledge base. This will allow us to give a semantic context or label for each visual feature. Our results show that in the aligned embedding space, nodes from the knowledge graph are close to the CNN feature nodes that have similar meanings.
  arXiv  Detail & Related papers  (2021-02-22T16:00:03Z)
• Streaming Graph Neural Networks via Continual Learning [31.810308087441445]
  Graph neural networks (GNNs) have achieved strong performance in various applications. In this paper, we propose a streaming GNN model based on continual learning. We show that our model can efficiently update model parameters and achieve comparable performance to model retraining.
  arXiv  Detail & Related papers  (2020-09-23T06:52:30Z)
• Explaining Deep Neural Networks using Unsupervised Clustering [12.639074798397619]
  We propose a novel method to explain trained deep neural networks (DNNs) by distilling them into surrogate models using unsupervised clustering. Our method can be applied flexibly to any subset of layers of a DNN architecture and can incorporate low-level and high-level information.
  arXiv  Detail & Related papers  (2020-07-15T04:49:43Z)
• Boosting Deep Neural Networks with Geometrical Prior Knowledge: A Survey [77.99182201815763]
  Deep Neural Networks (DNNs) achieve state-of-the-art results in many different problem settings. DNNs are often treated as black box systems, which complicates their evaluation and validation. One promising field, inspired by the success of convolutional neural networks (CNNs) in computer vision tasks, is to incorporate knowledge about symmetric geometrical transformations.
  arXiv  Detail & Related papers  (2020-06-30T14:56:05Z)
• Neural Additive Models: Interpretable Machine Learning with Neural Nets [77.66871378302774]
  Deep neural networks (DNNs) are powerful black-box predictors that have achieved impressive performance on a wide variety of tasks. We propose Neural Additive Models (NAMs) which combine some of the expressivity of DNNs with the inherent intelligibility of generalized additive models. NAMs learn a linear combination of neural networks that each attend to a single input feature.
  arXiv  Detail & Related papers  (2020-04-29T01:28:32Z)
• Architecture Disentanglement for Deep Neural Networks [174.16176919145377]
  We introduce neural architecture disentanglement (NAD) to explain the inner workings of deep neural networks (DNNs) NAD learns to disentangle a pre-trained DNN into sub-architectures according to independent tasks, forming information flows that describe the inference processes. Results show that misclassified images have a high probability of being assigned to task sub-architectures similar to the correct ones.
  arXiv  Detail & Related papers  (2020-03-30T08:34:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.