DEPARA: Deep Attribution Graph for Deep Knowledge Transferability

• URL: http://arxiv.org/abs/2003.07496v1
• Date: Tue, 17 Mar 2020 02:07:50 GMT
• Title: DEPARA: Deep Attribution Graph for Deep Knowledge Transferability
• Authors: Jie Song, Yixin Chen, Jingwen Ye, Xinchao Wang, Chengchao Shen, Feng Mao, Mingli Song
• Abstract summary: We propose the DEeP Attribution gRAph (DEPARA) to investigate the transferability of knowledge learned from PR-DNNs. In DEPARA, nodes correspond to the inputs and are represented by their vectorized attribution maps with regards to the outputs of the PR-DNN. The knowledge transferability of two PR-DNNs is measured by the similarity of their corresponding DEPARAs.
• Score: 91.06106524522237
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Exploring the intrinsic interconnections between the knowledge encoded in PRe-trained Deep Neural Networks (PR-DNNs) of heterogeneous tasks sheds light on their mutual transferability, and consequently enables knowledge transfer from one task to another so as to reduce the training effort of the latter. In this paper, we propose the DEeP Attribution gRAph (DEPARA) to investigate the transferability of knowledge learned from PR-DNNs. In DEPARA, nodes correspond to the inputs and are represented by their vectorized attribution maps with regards to the outputs of the PR-DNN. Edges denote the relatedness between inputs and are measured by the similarity of their features extracted from the PR-DNN. The knowledge transferability of two PR-DNNs is measured by the similarity of their corresponding DEPARAs. We apply DEPARA to two important yet under-studied problems in transfer learning: pre-trained model selection and layer selection. Extensive experiments are conducted to demonstrate the effectiveness and superiority of the proposed method in solving both these problems. Code, data and models reproducing the results in this paper are available at \url{https://github.com/zju-vipa/DEPARA}.

Related papers

• Supervised Gradual Machine Learning for Aspect Category Detection [0.9857683394266679]
  Aspect Category Detection (ACD) aims to identify implicit and explicit aspects in a given review sentence. We propose a novel approach to tackle the ACD task by combining Deep Neural Networks (DNNs) with Gradual Machine Learning (GML) in a supervised setting.
  arXiv  Detail & Related papers  (2024-04-08T07:21:46Z)
• Evaluating the structure of cognitive tasks with transfer learning [67.22168759751541]
  This study investigates the transferability of deep learning representations between different EEG decoding tasks. We conduct extensive experiments using state-of-the-art decoding models on two recently released EEG datasets.
  arXiv  Detail & Related papers  (2023-07-28T14:51:09Z)
• Auxiliary-Tasks Learning for Physics-Informed Neural Network-Based Partial Differential Equations Solving [13.196871939441273]
  Physics-informed neural networks (PINNs) have emerged as promising surrogate modes for solving partial differential equations (PDEs) We propose auxiliary-task learning-based ATL-PINNs, which provide four different auxiliary-task learning modes. Our findings show that the proposed auxiliary-task learning modes can significantly improve solution accuracy, achieving a maximum performance boost of 96.62%.
  arXiv  Detail & Related papers  (2023-07-12T13:46:40Z)
• Diverse and Faithful Knowledge-Grounded Dialogue Generation via Sequential Posterior Inference [82.28542500317445]
  We present an end-to-end learning framework, termed Sequential Posterior Inference (SPI), capable of selecting knowledge and generating dialogues. Unlike other methods, SPI does not require the inference network or assume a simple geometry of the posterior distribution.
  arXiv  Detail & Related papers  (2023-06-01T21:23:13Z)
• Poster: Link between Bias, Node Sensitivity and Long-Tail Distribution in trained DNNs [12.404169549562523]
  Training datasets with long-tail distribution pose a challenge for deep neural networks (DNNs) This work identifies the node bias that leads to a varying sensitivity of the nodes for different output classes. We support our reasoning using an empirical case study of the networks trained on a real-world dataset.
  arXiv  Detail & Related papers  (2023-03-29T10:49:31Z)
• Decomposing neural networks as mappings of correlation functions [57.52754806616669]
  We study the mapping between probability distributions implemented by a deep feed-forward network. We identify essential statistics in the data, as well as different information representations that can be used by neural networks.
  arXiv  Detail & Related papers  (2022-02-10T09:30:31Z)
• CRNNTL: convolutional recurrent neural network and transfer learning for QSAR modelling [4.090810719630087]
  We propose the convolutional recurrent neural network and transfer learning (CRNNTL) for QSAR modelling. Our strategy takes advantages of both convolutional and recurrent neural networks for feature extraction, as well as the data augmentation method.
  arXiv  Detail & Related papers  (2021-09-07T20:04:55Z)
• Shifu2: A Network Representation Learning Based Model for Advisor-advisee Relationship Mining [82.75996880087747]
  We propose a novel model based on Network Representation Learning (NRL), namely Shifu2. Shifu2 takes the collaboration network as input and the identified advisor-advisee relationship as output. We generate a large-scale academic genealogy dataset by taking advantage of Shifu2.
  arXiv  Detail & Related papers  (2020-08-17T05:40:06Z)
• Architecture Disentanglement for Deep Neural Networks [174.16176919145377]
  We introduce neural architecture disentanglement (NAD) to explain the inner workings of deep neural networks (DNNs) NAD learns to disentangle a pre-trained DNN into sub-architectures according to independent tasks, forming information flows that describe the inference processes. Results show that misclassified images have a high probability of being assigned to task sub-architectures similar to the correct ones.
  arXiv  Detail & Related papers  (2020-03-30T08:34:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.