Dense Sample Deep Learning
- URL: http://arxiv.org/abs/2307.10991v2
- Date: Fri, 21 Jul 2023 15:18:10 GMT
- Title: Dense Sample Deep Learning
- Authors: Stephen Jos\`e Hanson, Vivek Yadav, Catherine Hanson
- Abstract summary: Despite the growing use of Deep Learning (DL) networks, little is actually understood about the learning mechanisms and representations.
In this paper we explore these questions with a large (1.24M weights; VGG) DL in a novel high density sample task.
We harvest some basic observations of the learning dynamics of DL and propose a new theory of complex feature construction based on our results.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Deep Learning (DL) , a variant of the neural network algorithms originally
proposed in the 1980s, has made surprising progress in Artificial Intelligence
(AI), ranging from language translation, protein folding, autonomous cars, and
more recently human-like language models (CHATbots), all that seemed
intractable until very recently. Despite the growing use of Deep Learning (DL)
networks, little is actually understood about the learning mechanisms and
representations that makes these networks effective across such a diverse range
of applications. Part of the answer must be the huge scale of the architecture
and of course the large scale of the data, since not much has changed since
1987. But the nature of deep learned representations remain largely unknown.
Unfortunately training sets with millions or billions of tokens have unknown
combinatorics and Networks with millions or billions of hidden units cannot
easily be visualized and their mechanisms cannot be easily revealed. In this
paper, we explore these questions with a large (1.24M weights; VGG) DL in a
novel high density sample task (5 unique tokens with at minimum 500 exemplars
per token) which allows us to more carefully follow the emergence of category
structure and feature construction. We use various visualization methods for
following the emergence of the classification and the development of the
coupling of feature detectors and structures that provide a type of graphical
bootstrapping, From these results we harvest some basic observations of the
learning dynamics of DL and propose a new theory of complex feature
construction based on our results.
Related papers
- The Confusion is Real: GRAPHIC - A Network Science Approach to Confusion Matrices in Deep Learning [10.180939285053375]
GRAPHIC is an architecture-agnostic approach that analyzes neural networks on a class level.<n>GRAPHIC provides insights into linear class separability, dataset issues, and architectural behavior.<n>In summary, by uncovering real confusions, GRAPHIC offers new perspectives on how neural networks learn.
arXiv Detail & Related papers (2026-02-23T12:20:37Z) - Concept-Guided Interpretability via Neural Chunking [54.73787666584143]
We show that neural networks exhibit patterns in their raw population activity that mirror regularities in the training data.<n>We propose three methods to extract these emerging entities, complementing each other based on label availability and dimensionality.<n>Our work points to a new direction for interpretability, one that harnesses both cognitive principles and the structure of naturalistic data.
arXiv Detail & Related papers (2025-05-16T13:49:43Z) - Discovering Chunks in Neural Embeddings for Interpretability [53.80157905839065]
We propose leveraging the principle of chunking to interpret artificial neural population activities.
We first demonstrate this concept in recurrent neural networks (RNNs) trained on artificial sequences with imposed regularities.
We identify similar recurring embedding states corresponding to concepts in the input, with perturbations to these states activating or inhibiting the associated concepts.
arXiv Detail & Related papers (2025-02-03T20:30:46Z) - Towards Scalable and Versatile Weight Space Learning [51.78426981947659]
This paper introduces the SANE approach to weight-space learning.
Our method extends the idea of hyper-representations towards sequential processing of subsets of neural network weights.
arXiv Detail & Related papers (2024-06-14T13:12:07Z) - Unveiling the Unseen: Identifiable Clusters in Trained Depthwise
Convolutional Kernels [56.69755544814834]
Recent advances in depthwise-separable convolutional neural networks (DS-CNNs) have led to novel architectures.
This paper reveals another striking property of DS-CNN architectures: discernible and explainable patterns emerge in their trained depthwise convolutional kernels in all layers.
arXiv Detail & Related papers (2024-01-25T19:05:53Z) - Adinkra Symbol Recognition using Classical Machine Learning and Deep
Learning [0.0]
We build a CNN model for classification and recognition using six convolutional layers, three fully connected layers, and optional dropout regularization.
We assess the model's performance by measuring its accuracy and convergence rate.
We hope this application inspires ideas on the various uses of AI in organizing our traditional and modern lives.
arXiv Detail & Related papers (2023-11-27T11:26:41Z) - Interpretability of an Interaction Network for identifying $H
\rightarrow b\bar{b}$ jets [4.553120911976256]
In recent times, AI models based on deep neural networks are becoming increasingly popular for many of these applications.
We explore interpretability of AI models by examining an Interaction Network (IN) model designed to identify boosted $Hto bbarb$ jets.
We additionally illustrate the activity of hidden layers within the IN model as Neural Activation Pattern (NAP) diagrams.
arXiv Detail & Related papers (2022-11-23T08:38:52Z) - Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs.
By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
arXiv Detail & Related papers (2022-02-01T17:11:13Z) - Reasoning-Modulated Representations [85.08205744191078]
We study a common setting where our task is not purely opaque.
Our approach paves the way for a new class of data-efficient representation learning.
arXiv Detail & Related papers (2021-07-19T13:57:13Z) - Malicious Network Traffic Detection via Deep Learning: An Information
Theoretic View [0.0]
We study how homeomorphism affects learned representation of a malware traffic dataset.
Our results suggest that although the details of learned representations and the specific coordinate system defined over the manifold of all parameters differ slightly, the functional approximations are the same.
arXiv Detail & Related papers (2020-09-16T15:37:44Z) - Understanding the Role of Individual Units in a Deep Neural Network [85.23117441162772]
We present an analytic framework to systematically identify hidden units within image classification and image generation networks.
First, we analyze a convolutional neural network (CNN) trained on scene classification and discover units that match a diverse set of object concepts.
Second, we use a similar analytic method to analyze a generative adversarial network (GAN) model trained to generate scenes.
arXiv Detail & Related papers (2020-09-10T17:59:10Z) - AutoML-Zero: Evolving Machine Learning Algorithms From Scratch [76.83052807776276]
We show that it is possible to automatically discover complete machine learning algorithms just using basic mathematical operations as building blocks.
We demonstrate this by introducing a novel framework that significantly reduces human bias through a generic search space.
We believe these preliminary successes in discovering machine learning algorithms from scratch indicate a promising new direction in the field.
arXiv Detail & Related papers (2020-03-06T19:00:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.