Isometric Representations in Neural Networks Improve Robustness
- URL: http://arxiv.org/abs/2211.01236v1
- Date: Wed, 2 Nov 2022 16:18:18 GMT
- Title: Isometric Representations in Neural Networks Improve Robustness
- Authors: Kosio Beshkov, Jonas Verhellen and Mikkel Elle Lepper{\o}d
- Abstract summary: We train neural networks to perform classification while simultaneously maintaining within-class metric structure.
We verify that isometric regularization improves the robustness to adversarial attacks on MNIST.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Artificial and biological agents cannon learn given completely random and
unstructured data. The structure of data is encoded in the metric relationships
between data points. In the context of neural networks, neuronal activity
within a layer forms a representation reflecting the transformation that the
layer implements on its inputs. In order to utilize the structure in the data
in a truthful manner, such representations should reflect the input distances
and thus be continuous and isometric. Supporting this statement, recent
findings in neuroscience propose that generalization and robustness are tied to
neural representations being continuously differentiable. In machine learning,
most algorithms lack robustness and are generally thought to rely on aspects of
the data that differ from those that humans use, as is commonly seen in
adversarial attacks. During cross-entropy classification, the metric and
structural properties of network representations are usually broken both
between and within classes. This side effect from training can lead to
instabilities under perturbations near locations where such structure is not
preserved. One of the standard solutions to obtain robustness is to add ad hoc
regularization terms, but to our knowledge, forcing representations to preserve
the metric structure of the input data as a stabilising mechanism has not yet
been studied. In this work, we train neural networks to perform classification
while simultaneously maintaining within-class metric structure, leading to
isometric within-class representations. Such network representations turn out
to be beneficial for accurate and robust inference. By stacking layers with
this property we create a network architecture that facilitates hierarchical
manipulation of internal neural representations. Finally, we verify that
isometric regularization improves the robustness to adversarial attacks on
MNIST.
Related papers
- Coding schemes in neural networks learning classification tasks [52.22978725954347]
We investigate fully-connected, wide neural networks learning classification tasks.
We show that the networks acquire strong, data-dependent features.
Surprisingly, the nature of the internal representations depends crucially on the neuronal nonlinearity.
arXiv Detail & Related papers (2024-06-24T14:50:05Z) - Semantic Loss Functions for Neuro-Symbolic Structured Prediction [74.18322585177832]
We discuss the semantic loss, which injects knowledge about such structure, defined symbolically, into training.
It is agnostic to the arrangement of the symbols, and depends only on the semantics expressed thereby.
It can be combined with both discriminative and generative neural models.
arXiv Detail & Related papers (2024-05-12T22:18:25Z) - Formal Verification of Graph Convolutional Networks with Uncertain Node Features and Uncertain Graph Structure [7.133681867718039]
Graph neural networks are becoming increasingly popular in the field of machine learning.
They have been applied in safety-critical environments where perturbations inherently occur.
This research addresses the non-passing gap by preserving the dependencies of all elements in the underlying computations.
arXiv Detail & Related papers (2024-04-23T14:12:48Z) - Representational dissimilarity metric spaces for stochastic neural
networks [4.229248343585332]
Quantifying similarity between neural representations is a perennial problem in deep learning and neuroscience research.
We generalize shape metrics to quantify differences in representations.
We find that neurobiological oriented visual gratings and naturalistic scenes respectively resemble untrained and trained deep network representations.
arXiv Detail & Related papers (2022-11-21T17:32:40Z) - Do We Really Need a Learnable Classifier at the End of Deep Neural
Network? [118.18554882199676]
We study the potential of learning a neural network for classification with the classifier randomly as an ETF and fixed during training.
Our experimental results show that our method is able to achieve similar performances on image classification for balanced datasets.
arXiv Detail & Related papers (2022-03-17T04:34:28Z) - Interpretable part-whole hierarchies and conceptual-semantic
relationships in neural networks [4.153804257347222]
We present Agglomerator, a framework capable of providing a representation of part-whole hierarchies from visual cues.
We evaluate our method on common datasets, such as SmallNORB, MNIST, FashionMNIST, CIFAR-10, and CIFAR-100.
arXiv Detail & Related papers (2022-03-07T10:56:13Z) - Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs.
By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
arXiv Detail & Related papers (2022-02-01T17:11:13Z) - Dynamic Inference with Neural Interpreters [72.90231306252007]
We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules.
inputs to the model are routed through a sequence of functions in a way that is end-to-end learned.
We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
arXiv Detail & Related papers (2021-10-12T23:22:45Z) - Category-Learning with Context-Augmented Autoencoder [63.05016513788047]
Finding an interpretable non-redundant representation of real-world data is one of the key problems in Machine Learning.
We propose a novel method of using data augmentations when training autoencoders.
We train a Variational Autoencoder in such a way, that it makes transformation outcome predictable by auxiliary network.
arXiv Detail & Related papers (2020-10-10T14:04:44Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.