The QR decomposition for radial neural networks

• URL: http://arxiv.org/abs/2107.02550v1
• Date: Tue, 6 Jul 2021 11:41:02 GMT
• Title: The QR decomposition for radial neural networks
• Authors: Iordan Ganev, Robin Walters
• Abstract summary: We provide a theoretical framework for neural networks in terms of the representation theory of quivers. An exploitation of these symmetries leads to a model compression algorithm for radial neural networks.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We provide a theoretical framework for neural networks in terms of the representation theory of quivers, thus revealing symmetries of the parameter space of neural networks. An exploitation of these symmetries leads to a model compression algorithm for radial neural networks based on an analogue of the QR decomposition. A projected version of backpropogation on the original model matches usual backpropogation on the compressed model.

Related papers

• The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof [50.49582712378289]
  We investigate the impact of neural parameter symmetries by introducing new neural network architectures. We develop two methods, with some provable guarantees, of modifying standard neural networks to reduce parameter space symmetries. Our experiments reveal several interesting observations on the empirical impact of parameter symmetries.
  arXiv  Detail & Related papers  (2024-05-30T16:32:31Z)
• Novel Kernel Models and Exact Representor Theory for Neural Networks Beyond the Over-Parameterized Regime [52.00917519626559]
  This paper presents two models of neural-networks and their training applicable to neural networks of arbitrary width, depth and topology. We also present an exact novel representor theory for layer-wise neural network training with unregularized gradient descent in terms of a local-extrinsic neural kernel (LeNK) This representor theory gives insight into the role of higher-order statistics in neural network training and the effect of kernel evolution in neural-network kernel models.
  arXiv  Detail & Related papers  (2024-05-24T06:30:36Z)
• Graph Neural Networks for Learning Equivariant Representations of Neural Networks [55.04145324152541]
  We propose to represent neural networks as computational graphs of parameters. Our approach enables a single model to encode neural computational graphs with diverse architectures. We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations.
  arXiv  Detail & Related papers  (2024-03-18T18:01:01Z)
• Unification of Symmetries Inside Neural Networks: Transformer, Feedforward and Neural ODE [2.002741592555996]
  This study introduces a novel approach by applying the principles of gauge symmetries, a key concept in physics, to neural network architectures. We mathematically formulate the parametric redundancies in neural ODEs, and find that their gauge symmetries are given by spacetime diffeomorphisms. Viewing neural ODEs as a continuum version of feedforward neural networks, we show that the parametric redundancies in feedforward neural networks are indeed lifted to diffeomorphisms in neural ODEs.
  arXiv  Detail & Related papers  (2024-02-04T06:11:54Z)
• A predictive physics-aware hybrid reduced order model for reacting flows [65.73506571113623]
  A new hybrid predictive Reduced Order Model (ROM) is proposed to solve reacting flow problems. The number of degrees of freedom is reduced from thousands of temporal points to a few POD modes with their corresponding temporal coefficients. Two different deep learning architectures have been tested to predict the temporal coefficients.
  arXiv  Detail & Related papers  (2023-01-24T08:39:20Z)
• Interrelation of equivariant Gaussian processes and convolutional neural networks [77.34726150561087]
  Currently there exists rather promising new trend in machine leaning (ML) based on the relationship between neural networks (NN) and Gaussian processes (GP) In this work we establish a relationship between the many-channel limit for CNNs equivariant with respect to two-dimensional Euclidean group with vector-valued neuron activations and the corresponding independently introduced equivariant Gaussian processes (GP)
  arXiv  Detail & Related papers  (2022-09-17T17:02:35Z)
• Quiver neural networks [5.076419064097734]
  We develop a uniform theoretical approach towards the analysis of various neural network connectivity architectures. Inspired by quiver representation theory in mathematics, this approach gives a compact way to capture elaborate data flows.
  arXiv  Detail & Related papers  (2022-07-26T09:42:45Z)
• Robust lEarned Shrinkage-Thresholding (REST): Robust unrolling for sparse recover [87.28082715343896]
  We consider deep neural networks for solving inverse problems that are robust to forward model mis-specifications. We design a new robust deep neural network architecture by applying algorithm unfolding techniques to a robust version of the underlying recovery problem. The proposed REST network is shown to outperform state-of-the-art model-based and data-driven algorithms in both compressive sensing and radar imaging problems.
  arXiv  Detail & Related papers  (2021-10-20T06:15:45Z)
• Invertible Neural Networks versus MCMC for Posterior Reconstruction in Grazing Incidence X-Ray Fluorescence [0.3232625980782302]
  We propose to reconstruct the posterior parameter distribution given a noisy measurement generated by the forward model by an appropriately learned invertible neural network. We demonstrate by numerical comparisons that our method can compete with established Markov Chain Monte Carlo approaches, while being more efficient and flexible in applications.
  arXiv  Detail & Related papers  (2021-02-05T14:17:59Z)
• Ridge Regression with Over-Parametrized Two-Layer Networks Converge to Ridgelet Spectrum [10.05944106581306]
  We show that the distribution of the parameters converges to a spectrum of the ridgelet transform. This result provides a new insight into the characterization of the local minima of neural networks.
  arXiv  Detail & Related papers  (2020-07-07T13:47:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.