A Quantitative Approach to Predicting Representational Learning and Performance in Neural Networks

• URL: http://arxiv.org/abs/2307.07575v1
• Date: Fri, 14 Jul 2023 18:39:04 GMT
• Title: A Quantitative Approach to Predicting Representational Learning and Performance in Neural Networks
• Authors: Ryan Pyle, Sebastian Musslick, Jonathan D. Cohen, and Ankit B. Patel
• Abstract summary: Key property of neural networks is how they learn to represent and manipulate input information in order to solve a task. We introduce a new pseudo-kernel based tool for analyzing and predicting learned representations.
• Score: 5.544128024203989
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: A key property of neural networks (both biological and artificial) is how they learn to represent and manipulate input information in order to solve a task. Different types of representations may be suited to different types of tasks, making identifying and understanding learned representations a critical part of understanding and designing useful networks. In this paper, we introduce a new pseudo-kernel based tool for analyzing and predicting learned representations, based only on the initial conditions of the network and the training curriculum. We validate the method on a simple test case, before demonstrating its use on a question about the effects of representational learning on sequential single versus concurrent multitask performance. We show that our method can be used to predict the effects of the scale of weight initialization and training curriculum on representational learning and downstream concurrent multitasking performance.

Related papers

• Towards Scalable and Versatile Weight Space Learning [51.78426981947659]
  This paper introduces the SANE approach to weight-space learning. Our method extends the idea of hyper-representations towards sequential processing of subsets of neural network weights.
  arXiv  Detail & Related papers  (2024-06-14T13:12:07Z)
• Proto-Value Networks: Scaling Representation Learning with Auxiliary Tasks [33.98624423578388]
  Auxiliary tasks improve representations learned by deep reinforcement learning agents. We derive a new family of auxiliary tasks based on the successor measure. We show that proto-value networks produce rich features that may be used to obtain performance comparable to established algorithms.
  arXiv  Detail & Related papers  (2023-04-25T04:25:08Z)
• Complexity of Representations in Deep Learning [2.0219767626075438]
  We analyze the effectiveness of the learned representations in separating the classes from a data complexity perspective. We show how the data complexity evolves through the network, how it changes during training, and how it is impacted by the network design and the availability of training samples.
  arXiv  Detail & Related papers  (2022-09-01T15:20:21Z)
• Active Multi-Task Representation Learning [50.13453053304159]
  We give the first formal study on resource task sampling by leveraging the techniques from active learning. We propose an algorithm that iteratively estimates the relevance of each source task to the target task and samples from each source task based on the estimated relevance.
  arXiv  Detail & Related papers  (2022-02-02T08:23:24Z)
• On the relationship between disentanglement and multi-task learning [62.997667081978825]
  We take a closer look at the relationship between disentanglement and multi-task learning based on hard parameter sharing. We show that disentanglement appears naturally during the process of multi-task neural network training.
  arXiv  Detail & Related papers  (2021-10-07T14:35:34Z)
• Explaining Deep Learning Representations by Tracing the Training Process [10.774699463547439]
  We propose a novel explanation method that explains the decisions of a deep neural network. We investigate how the intermediate representations at each layer of the deep network were refined during the training process. We show that our method identifies highly representative training instances that can be used as an explanation.
  arXiv  Detail & Related papers  (2021-09-13T11:29:04Z)
• Multivariate Business Process Representation Learning utilizing Gramian Angular Fields and Convolutional Neural Networks [0.0]
  Learning meaningful representations of data is an important aspect of machine learning. For predictive process analytics, it is essential to have all explanatory characteristics of a process instance available. We propose a novel approach for representation learning of business process instances.
  arXiv  Detail & Related papers  (2021-06-15T10:21:14Z)
• Explainability-aided Domain Generalization for Image Classification [0.0]
  We show that applying methods and architectures from the explainability literature can achieve state-of-the-art performance for the challenging task of domain generalization. We develop a set of novel algorithms including DivCAM, an approach where the network receives guidance during training via gradient based class activation maps to focus on a diverse set of discriminative features. Since these methods offer competitive performance on top of explainability, we argue that the proposed methods can be used as a tool to improve the robustness of deep neural network architectures.
  arXiv  Detail & Related papers  (2021-04-05T02:27:01Z)
• Usable Information and Evolution of Optimal Representations During Training [79.38872675793813]
  In particular, we find that semantically meaningful but ultimately irrelevant information is encoded in the early transient dynamics of training. We show these effects on both perceptual decision-making tasks inspired by literature, as well as on standard image classification tasks.
  arXiv  Detail & Related papers  (2020-10-06T03:50:19Z)
• Region Comparison Network for Interpretable Few-shot Image Classification [97.97902360117368]
  Few-shot image classification has been proposed to effectively use only a limited number of labeled examples to train models for new classes. We propose a metric learning based method named Region Comparison Network (RCN), which is able to reveal how few-shot learning works. We also present a new way to generalize the interpretability from the level of tasks to categories.
  arXiv  Detail & Related papers  (2020-09-08T07:29:05Z)
• Pre-training Text Representations as Meta Learning [113.3361289756749]
  We introduce a learning algorithm which directly optimize model's ability to learn text representations for effective learning of downstream tasks. We show that there is an intrinsic connection between multi-task pre-training and model-agnostic meta-learning with a sequence of meta-train steps.
  arXiv  Detail & Related papers  (2020-04-12T09:05:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.