Rethink Deep Learning with Invariance in Data Representation

• URL: http://arxiv.org/abs/2412.04858v1
• Date: Fri, 06 Dec 2024 08:52:26 GMT
• Title: Rethink Deep Learning with Invariance in Data Representation
• Authors: Shuren Qi, Fei Wang, Tieyong Zeng, Fenglei Fan,
• Abstract summary: Invariant design has been the cornerstone of various representations in the era before deep learning. In this tutorial, we will give a historical perspective of the invariance in data representations. We will identify those research dilemmas, promising works, future directions, and web applications.
• Score: 23.49898692565483
• License:
• Abstract: Integrating invariance into data representations is a principled design in intelligent systems and web applications. Representations play a fundamental role, where systems and applications are both built on meaningful representations of digital inputs (rather than the raw data). In fact, the proper design/learning of such representations relies on priors w.r.t. the task of interest. Here, the concept of symmetry from the Erlangen Program may be the most fruitful prior -- informally, a symmetry of a system is a transformation that leaves a certain property of the system invariant. Symmetry priors are ubiquitous, e.g., translation as a symmetry of the object classification, where object category is invariant under translation. The quest for invariance is as old as pattern recognition and data mining itself. Invariant design has been the cornerstone of various representations in the era before deep learning, such as the SIFT. As we enter the early era of deep learning, the invariance principle is largely ignored and replaced by a data-driven paradigm, such as the CNN. However, this neglect did not last long before they encountered bottlenecks regarding robustness, interpretability, efficiency, and so on. The invariance principle has returned in the era of rethinking deep learning, forming a new field known as Geometric Deep Learning (GDL). In this tutorial, we will give a historical perspective of the invariance in data representations. More importantly, we will identify those research dilemmas, promising works, future directions, and web applications.

Related papers

• Unsupervised Representation Learning from Sparse Transformation Analysis [79.94858534887801]
  We propose to learn representations from sequence data by factorizing the transformations of the latent variables into sparse components. Input data are first encoded as distributions of latent activations and subsequently transformed using a probability flow model.
  arXiv  Detail & Related papers  (2024-10-07T23:53:25Z)
• Deep networks for system identification: a Survey [56.34005280792013]
  System identification learns mathematical descriptions of dynamic systems from input-output data. Main aim of the identified model is to predict new data from previous observations. We discuss architectures commonly adopted in the literature, like feedforward, convolutional, and recurrent networks.
  arXiv  Detail & Related papers  (2023-01-30T12:38:31Z)
• In What Ways Are Deep Neural Networks Invariant and How Should We Measure This? [5.757836174655293]
  We introduce a family of invariance and equivariance metrics that allows us to quantify these properties in a way that disentangles them from other metrics such as loss or accuracy. We draw a range of conclusions about invariance and equivariance in deep learning models, ranging from whether initializing a model with pretrained weights has an effect on a trained model's invariance, to the extent to which invariance learned via training can generalize to out-of-distribution data.
  arXiv  Detail & Related papers  (2022-10-07T18:43:21Z)
• The Lie Derivative for Measuring Learned Equivariance [84.29366874540217]
  We study the equivariance properties of hundreds of pretrained models, spanning CNNs, transformers, and Mixer architectures. We find that many violations of equivariance can be linked to spatial aliasing in ubiquitous network layers, such as pointwise non-linearities. For example, transformers can be more equivariant than convolutional neural networks after training.
  arXiv  Detail & Related papers  (2022-10-06T15:20:55Z)
• Learning sparse features can lead to overfitting in neural networks [9.2104922520782]
  We show that feature learning can perform worse than lazy training. Although sparsity is known to be essential for learning anisotropic data, it is detrimental when the target function is constant or smooth.
  arXiv  Detail & Related papers  (2022-06-24T14:26:33Z)
• Fair Interpretable Learning via Correction Vectors [68.29997072804537]
  We propose a new framework for fair representation learning centered around the learning of "correction vectors" The corrections are then simply summed up to the original features, and can therefore be analyzed as an explicit penalty or bonus to each feature. We show experimentally that a fair representation learning problem constrained in such a way does not impact performance.
  arXiv  Detail & Related papers  (2022-01-17T10:59:33Z)
• Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training [4.318555434063275]
  Deep learning algorithms are responsible for a technological revolution in a variety of tasks including image recognition or Go playing. Yet, why they work is not understood. Ultimately, they manage to classify data lying in high dimension -- a feat generically impossible. We argue that different learning regimes can be organized into a phase diagram.
  arXiv  Detail & Related papers  (2020-12-30T11:00:36Z)
• Learning Invariances in Neural Networks [51.20867785006147]
  We show how to parameterize a distribution over augmentations and optimize the training loss simultaneously with respect to the network parameters and augmentation parameters. We can recover the correct set and extent of invariances on image classification, regression, segmentation, and molecular property prediction from a large space of augmentations.
  arXiv  Detail & Related papers  (2020-10-22T17:18:48Z)
• Category-Learning with Context-Augmented Autoencoder [63.05016513788047]
  Finding an interpretable non-redundant representation of real-world data is one of the key problems in Machine Learning. We propose a novel method of using data augmentations when training autoencoders. We train a Variational Autoencoder in such a way, that it makes transformation outcome predictable by auxiliary network.
  arXiv  Detail & Related papers  (2020-10-10T14:04:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.