Breaking the Curse of Dimensionality in Deep Neural Networks by Learning Invariant Representations

• URL: http://arxiv.org/abs/2310.16154v1
• Date: Tue, 24 Oct 2023 19:50:41 GMT
• Title: Breaking the Curse of Dimensionality in Deep Neural Networks by Learning Invariant Representations
• Authors: Leonardo Petrini
• Abstract summary: This thesis explores the theoretical foundations of deep learning by studying the relationship between the architecture of these models and the inherent structures found within the data they process. We ask What drives the efficacy of deep learning algorithms and allows them to beat the so-called curse of dimensionality. Our methodology takes an empirical approach to deep learning, combining experimental studies with physics-inspired toy models.
• Score: 1.9580473532948401
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Artificial intelligence, particularly the subfield of machine learning, has seen a paradigm shift towards data-driven models that learn from and adapt to data. This has resulted in unprecedented advancements in various domains such as natural language processing and computer vision, largely attributed to deep learning, a special class of machine learning models. Deep learning arguably surpasses traditional approaches by learning the relevant features from raw data through a series of computational layers. This thesis explores the theoretical foundations of deep learning by studying the relationship between the architecture of these models and the inherent structures found within the data they process. In particular, we ask What drives the efficacy of deep learning algorithms and allows them to beat the so-called curse of dimensionality-i.e. the difficulty of generally learning functions in high dimensions due to the exponentially increasing need for data points with increased dimensionality? Is it their ability to learn relevant representations of the data by exploiting their structure? How do different architectures exploit different data structures? In order to address these questions, we push forward the idea that the structure of the data can be effectively characterized by its invariances-i.e. aspects that are irrelevant for the task at hand. Our methodology takes an empirical approach to deep learning, combining experimental studies with physics-inspired toy models. These simplified models allow us to investigate and interpret the complex behaviors we observe in deep learning systems, offering insights into their inner workings, with the far-reaching goal of bridging the gap between theory and practice.

Related papers

• A Survey on State-of-the-art Deep Learning Applications and Challenges [0.0]
  Building a deep learning model is challenging due to the algorithm's complexity and the dynamic nature of real-world problems. This study aims to comprehensively review the state-of-the-art deep learning models in computer vision, natural language processing, time series analysis and pervasive computing.
  arXiv  Detail & Related papers  (2024-03-26T10:10:53Z)
• Homological Convolutional Neural Networks [4.615338063719135]
  We propose a novel deep learning architecture that exploits the data structural organization through topologically constrained network representations. We test our model on 18 benchmark datasets against 5 classic machine learning and 3 deep learning models.
  arXiv  Detail & Related papers  (2023-08-26T08:48:51Z)
• Deep networks for system identification: a Survey [56.34005280792013]
  System identification learns mathematical descriptions of dynamic systems from input-output data. Main aim of the identified model is to predict new data from previous observations. We discuss architectures commonly adopted in the literature, like feedforward, convolutional, and recurrent networks.
  arXiv  Detail & Related papers  (2023-01-30T12:38:31Z)
• From Actions to Events: A Transfer Learning Approach Using Improved Deep Belief Networks [1.0554048699217669]
  This paper proposes a novel approach to map the knowledge from action recognition to event recognition using an energy-based model. Such a model can process all frames simultaneously, carrying spatial and temporal information through the learning process.
  arXiv  Detail & Related papers  (2022-11-30T14:47:10Z)
• The Neural Race Reduction: Dynamics of Abstraction in Gated Networks [12.130628846129973]
  We introduce the Gated Deep Linear Network framework that schematizes how pathways of information flow impact learning dynamics. We derive an exact reduction and, for certain cases, exact solutions to the dynamics of learning. Our work gives rise to general hypotheses relating neural architecture to learning and provides a mathematical approach towards understanding the design of more complex architectures.
  arXiv  Detail & Related papers  (2022-07-21T12:01:03Z)
• Tensor Methods in Computer Vision and Deep Learning [120.3881619902096]
  tensors, or multidimensional arrays, are data structures that can naturally represent visual data of multiple dimensions. With the advent of the deep learning paradigm shift in computer vision, tensors have become even more fundamental. This article provides an in-depth and practical review of tensors and tensor methods in the context of representation learning and deep learning.
  arXiv  Detail & Related papers  (2021-07-07T18:42:45Z)
• Ten Quick Tips for Deep Learning in Biology [116.78436313026478]
  Machine learning is concerned with the development and applications of algorithms that can recognize patterns in data and use them for predictive modeling. Deep learning has become its own subfield of machine learning. In the context of biological research, deep learning has been increasingly used to derive novel insights from high-dimensional biological data.
  arXiv  Detail & Related papers  (2021-05-29T21:02:44Z)
• A neural anisotropic view of underspecification in deep learning [60.119023683371736]
  We show that the way neural networks handle the underspecification of problems is highly dependent on the data representation. Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.
  arXiv  Detail & Related papers  (2021-04-29T14:31:09Z)
• Knowledge as Invariance -- History and Perspectives of Knowledge-augmented Machine Learning [69.99522650448213]
  Research in machine learning is at a turning point. Research interests are shifting away from increasing the performance of highly parameterized models to exceedingly specific tasks. This white paper provides an introduction and discussion of this emerging field in machine learning research.
  arXiv  Detail & Related papers  (2020-12-21T15:07:19Z)
• Model-Based Deep Learning [155.063817656602]
  Signal processing, communications, and control have traditionally relied on classical statistical modeling techniques. Deep neural networks (DNNs) use generic architectures which learn to operate from data, and demonstrate excellent performance. We are interested in hybrid techniques that combine principled mathematical models with data-driven systems to benefit from the advantages of both approaches.
  arXiv  Detail & Related papers  (2020-12-15T16:29:49Z)
• Malicious Network Traffic Detection via Deep Learning: An Information Theoretic View [0.0]
  We study how homeomorphism affects learned representation of a malware traffic dataset. Our results suggest that although the details of learned representations and the specific coordinate system defined over the manifold of all parameters differ slightly, the functional approximations are the same.
  arXiv  Detail & Related papers  (2020-09-16T15:37:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.