Functional Equivalence and Path Connectivity of Reducible Hyperbolic Tangent Networks

• URL: http://arxiv.org/abs/2305.05089v2
• Date: Wed, 7 Jun 2023 06:05:18 GMT
• Title: Functional Equivalence and Path Connectivity of Reducible Hyperbolic Tangent Networks
• Abstract summary: A neural network parameter's functional equivalence class is the set of parameters implementing the same input--output function. We give an algorithmic characterisation of unit redundancies and reducible functional equivalence classes for a single-hidden-layer hyperbolic tangent architecture.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Understanding the learning process of artificial neural networks requires clarifying the structure of the parameter space within which learning takes place. A neural network parameter's functional equivalence class is the set of parameters implementing the same input--output function. For many architectures, almost all parameters have a simple and well-documented functional equivalence class. However, there is also a vanishing minority of reducible parameters, with richer functional equivalence classes caused by redundancies among the network's units. In this paper, we give an algorithmic characterisation of unit redundancies and reducible functional equivalence classes for a single-hidden-layer hyperbolic tangent architecture. We show that such functional equivalence classes are piecewise-linear path-connected sets, and that for parameters with a majority of redundant units, the sets have a diameter of at most 7 linear segments.

Related papers

• Most ReLU Networks Admit Identifiable Parameters [14.626408075188005]
  We study the realization map of deep ReLU networks, focusing on when a function determines its parameters up to scaling and permutation.<n>Our main result shows that for every architecture whose input and hidden layers have width at least two, there exists an open set of identifiable parameters.
  arXiv  Detail & Related papers  (2026-05-05T10:25:28Z)
• A Complete Symmetry Classification of Shallow ReLU Networks [0.0]
  We study the study of distinct parameters in neural architectures which realize the same function.<n>We exploit the non-differentiability of the ReLU activation to provide a complete classification of the symmetries in the shallow case.
  arXiv  Detail & Related papers  (2026-04-15T16:18:44Z)
• Element-wise Modulation of Random Matrices for Efficient Neural Layers [0.0]
  We propose a novel approach that decouples feature mixing from adaptation by utilizing a fixed random matrix modulated by lightweight, learnable element-wise parameters.<n>This architecture drastically reduces the trainable parameter count to a linear scale while retaining reliable accuracy across various benchmarks.
  arXiv  Detail & Related papers  (2025-12-15T16:16:53Z)
• Structurally Flexible Neural Networks: Evolving the Building Blocks for General Agents [7.479827648985631]
  We show that it is possible to optimize a set of distinct neuron- and synapse types allowing for a mitigation of the symmetry dilemma. We demonstrate this by optimizing a single set of neurons and synapses to solve multiple reinforcement learning control tasks simultaneously.
  arXiv  Detail & Related papers  (2024-04-06T14:04:14Z)
• Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory [64.06519549649495]
  We provide the definition of what we call functionally equivalent features. These features produce equivalent output under certain transformations. We propose an efficient algorithm named Iterative Feature Merging.
  arXiv  Detail & Related papers  (2023-10-10T16:27:12Z)
• Neural approximation of Wasserstein distance via a universal architecture for symmetric and factorwise group invariant functions [6.994580267603235]
  We first present a general neural network architecture for approximating SFGI functions. The main contribution of this paper combines this general neural network with a sketching idea to develop a specific and efficient neural network. Our work provides an interesting integration of sketching ideas for geometric problems with universal approximation of symmetric functions.
  arXiv  Detail & Related papers  (2023-08-01T04:11:19Z)
• ReLU Neural Networks with Linear Layers are Biased Towards Single- and Multi-Index Models [9.96121040675476]
  This manuscript explores how properties of functions learned by neural networks of depth greater than two layers affect predictions. Our framework considers a family of networks of varying depths that all have the same capacity but different representation costs.
  arXiv  Detail & Related papers  (2023-05-24T22:10:12Z)
• Equivariance with Learned Canonicalization Functions [77.32483958400282]
  We show that learning a small neural network to perform canonicalization is better than using predefineds. Our experiments show that learning the canonicalization function is competitive with existing techniques for learning equivariant functions across many tasks.
  arXiv  Detail & Related papers  (2022-11-11T21:58:15Z)
• Stabilizing Q-learning with Linear Architectures for Provably Efficient Learning [53.17258888552998]
  This work proposes an exploration variant of the basic $Q$-learning protocol with linear function approximation. We show that the performance of the algorithm degrades very gracefully under a novel and more permissive notion of approximation error.
  arXiv  Detail & Related papers  (2022-06-01T23:26:51Z)
• Parametric Rectified Power Sigmoid Units: Learning Nonlinear Neural Transfer Analytical Forms [1.6975704972827304]
  The paper proposes representation functionals in a dual paradigm where learning jointly concerns linear convolutional weights and parametric forms of nonlinear activation functions. The nonlinear forms proposed for performing the functional representation are associated with a new class of parametric neural transfer functions called rectified power sigmoid units. Performance achieved by the joint learning of convolutional and rectified power sigmoid learnable parameters are shown outstanding in both shallow and deep learning frameworks.
  arXiv  Detail & Related papers  (2021-01-25T08:25:22Z)
• Multi-task Supervised Learning via Cross-learning [102.64082402388192]
  We consider a problem known as multi-task learning, consisting of fitting a set of regression functions intended for solving different tasks. In our novel formulation, we couple the parameters of these functions, so that they learn in their task specific domains while staying close to each other. This facilitates cross-fertilization in which data collected across different domains help improving the learning performance at each other task.
  arXiv  Detail & Related papers  (2020-10-24T21:35:57Z)
• A Functional Perspective on Learning Symmetric Functions with Neural Networks [48.80300074254758]
  We study the learning and representation of neural networks defined on measures. We establish approximation and generalization bounds under different choices of regularization. The resulting models can be learned efficiently and enjoy generalization guarantees that extend across input sizes.
  arXiv  Detail & Related papers  (2020-08-16T16:34:33Z)
• Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach [144.21892195917758]
  We study estimation in a class of generalized Structural equation models (SEMs) We formulate the linear operator equation as a min-max game, where both players are parameterized by neural networks (NNs), and learn the parameters of these neural networks using a gradient descent. For the first time we provide a tractable estimation procedure for SEMs based on NNs with provable convergence and without the need for sample splitting.
  arXiv  Detail & Related papers  (2020-07-02T17:55:47Z)
• No one-hidden-layer neural network can represent multivariable functions [0.0]
  In a function approximation with a neural network, an input dataset is mapped to an output index by optimizing the parameters of each hidden-layer unit. We present constraints on the parameters and its second derivative by constructing a continuum version of a one-hidden-layer neural network with the rectified linear unit (ReLU) activation function.
  arXiv  Detail & Related papers  (2020-06-19T06:46:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.