Implicitly Defined Layers in Neural Networks

• URL: http://arxiv.org/abs/2003.01822v2
• Date: Wed, 3 Jun 2020 01:36:06 GMT
• Title: Implicitly Defined Layers in Neural Networks
• Authors: Qianggong Zhang, Yanyang Gu, Michalkiewicz Mateusz, Mahsa Baktashmotlagh, Anders Eriksson
• Abstract summary: We show that defining individual layers in a neural network emphimplicitly provide much richer representations over the standard explicit one. We show how implicitly defined layers can be seamlessly incorporated into existing machine learning libraries.
• Score: 12.167549413547908
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In conventional formulations of multilayer feedforward neural networks, the individual layers are customarily defined by explicit functions. In this paper we demonstrate that defining individual layers in a neural network \emph{implicitly} provide much richer representations over the standard explicit one, consequently enabling a vastly broader class of end-to-end trainable architectures. We present a general framework of implicitly defined layers, where much of the theoretical analysis of such layers can be addressed through the implicit function theorem. We also show how implicitly defined layers can be seamlessly incorporated into existing machine learning libraries. In particular with respect to current automatic differentiation techniques for use in backpropagation based training. Finally, we demonstrate the versatility and relevance of our proposed approach on a number of diverse example problems with promising results.

Related papers

• WLD-Reg: A Data-dependent Within-layer Diversity Regularizer [98.78384185493624]
  Neural networks are composed of multiple layers arranged in a hierarchical structure jointly trained with a gradient-based optimization. We propose to complement this traditional 'between-layer' feedback with additional 'within-layer' feedback to encourage the diversity of the activations within the same layer. We present an extensive empirical study confirming that the proposed approach enhances the performance of several state-of-the-art neural network models in multiple tasks.
  arXiv  Detail & Related papers  (2023-01-03T20:57:22Z)
• Multilevel-in-Layer Training for Deep Neural Network Regression [1.6185544531149159]
  We present a multilevel regularization strategy that constructs and trains a hierarchy of neural networks. We experimentally show with PDE regression problems that our multilevel training approach is an effective regularizer.
  arXiv  Detail & Related papers  (2022-11-11T23:53:46Z)
• On skip connections and normalisation layers in deep optimisation [32.51139594406463]
  We introduce a general theoretical framework for the study of optimisation of deep neural networks. Our framework determines the curvature and regularity properties of multilayer loss landscapes. We identify a novel causal mechanism by which skip connections accelerate training.
  arXiv  Detail & Related papers  (2022-10-10T06:22:46Z)
• Neural network layers as parametric spans [0.0]
  We present a general definition of linear layer arising from a categorical framework based on the notions of integration theory and parametric spans. This definition generalizes and encompasses classical layers (e.g., dense, convolutional) while guaranteeing existence and computability of the layer's derivatives for backpropagation.
  arXiv  Detail & Related papers  (2022-08-01T12:41:22Z)
• Rank Diminishing in Deep Neural Networks [71.03777954670323]
  Rank of neural networks measures information flowing across layers. It is an instance of a key structural condition that applies across broad domains of machine learning. For neural networks, however, the intrinsic mechanism that yields low-rank structures remains vague and unclear.
  arXiv  Detail & Related papers  (2022-06-13T12:03:32Z)
• Interpretable part-whole hierarchies and conceptual-semantic relationships in neural networks [4.153804257347222]
  We present Agglomerator, a framework capable of providing a representation of part-whole hierarchies from visual cues. We evaluate our method on common datasets, such as SmallNORB, MNIST, FashionMNIST, CIFAR-10, and CIFAR-100.
  arXiv  Detail & Related papers  (2022-03-07T10:56:13Z)
• Defensive Tensorization [113.96183766922393]
  We propose tensor defensiveization, an adversarial defence technique that leverages a latent high-order factorization of the network. We empirically demonstrate the effectiveness of our approach on standard image classification benchmarks. We validate the versatility of our approach across domains and low-precision architectures by considering an audio task and binary networks.
  arXiv  Detail & Related papers  (2021-10-26T17:00:16Z)
• Dynamic Inference with Neural Interpreters [72.90231306252007]
  We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules. inputs to the model are routed through a sequence of functions in a way that is end-to-end learned. We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
  arXiv  Detail & Related papers  (2021-10-12T23:22:45Z)
• Explainability-aided Domain Generalization for Image Classification [0.0]
  We show that applying methods and architectures from the explainability literature can achieve state-of-the-art performance for the challenging task of domain generalization. We develop a set of novel algorithms including DivCAM, an approach where the network receives guidance during training via gradient based class activation maps to focus on a diverse set of discriminative features. Since these methods offer competitive performance on top of explainability, we argue that the proposed methods can be used as a tool to improve the robustness of deep neural network architectures.
  arXiv  Detail & Related papers  (2021-04-05T02:27:01Z)
• Dual-constrained Deep Semi-Supervised Coupled Factorization Network with Enriched Prior [80.5637175255349]
  We propose a new enriched prior based Dual-constrained Deep Semi-Supervised Coupled Factorization Network, called DS2CF-Net. To ex-tract hidden deep features, DS2CF-Net is modeled as a deep-structure and geometrical structure-constrained neural network. Our network can obtain state-of-the-art performance for representation learning and clustering.
  arXiv  Detail & Related papers  (2020-09-08T13:10:21Z)
• Automated Search for Resource-Efficient Branched Multi-Task Networks [81.48051635183916]
  We propose a principled approach, rooted in differentiable neural architecture search, to automatically define branching structures in a multi-task neural network. We show that our approach consistently finds high-performing branching structures within limited resource budgets.
  arXiv  Detail & Related papers  (2020-08-24T09:49:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.