NeRN -- Learning Neural Representations for Neural Networks

• URL: http://arxiv.org/abs/2212.13554v2
• Date: Fri, 21 Apr 2023 15:25:39 GMT
• Title: NeRN -- Learning Neural Representations for Neural Networks
• Authors: Maor Ashkenazi, Zohar Rimon, Ron Vainshtein, Shir Levi, Elad Richardson, Pinchas Mintz, Eran Treister
• Abstract summary: We show that, when adapted correctly, neural representations can be used to represent the weights of a pre-trained convolutional neural network. Inspired by coordinate inputs of previous neural representation methods, we assign a coordinate to each convolutional kernel in our network. We present two applications using NeRN, demonstrating the capabilities of the learned representations.
• Score: 3.7384109981836153
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural Representations have recently been shown to effectively reconstruct a wide range of signals from 3D meshes and shapes to images and videos. We show that, when adapted correctly, neural representations can be used to directly represent the weights of a pre-trained convolutional neural network, resulting in a Neural Representation for Neural Networks (NeRN). Inspired by coordinate inputs of previous neural representation methods, we assign a coordinate to each convolutional kernel in our network based on its position in the architecture, and optimize a predictor network to map coordinates to their corresponding weights. Similarly to the spatial smoothness of visual scenes, we show that incorporating a smoothness constraint over the original network's weights aids NeRN towards a better reconstruction. In addition, since slight perturbations in pre-trained model weights can result in a considerable accuracy loss, we employ techniques from the field of knowledge distillation to stabilize the learning process. We demonstrate the effectiveness of NeRN in reconstructing widely used architectures on CIFAR-10, CIFAR-100, and ImageNet. Finally, we present two applications using NeRN, demonstrating the capabilities of the learned representations.

Related papers

• Graph Neural Networks for Learning Equivariant Representations of Neural Networks [55.04145324152541]
  We propose to represent neural networks as computational graphs of parameters. Our approach enables a single model to encode neural computational graphs with diverse architectures. We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations.
  arXiv  Detail & Related papers  (2024-03-18T18:01:01Z)
• NeRD: Neural field-based Demosaicking [10.791425064370511]
  NeRD is a new demosaicking method for generating full-color images from Bayer patterns. We leverage advancements in neural fields to perform demosaicking by representing an image as a coordinate-based neural network with sine activation functions.
  arXiv  Detail & Related papers  (2023-04-13T14:25:05Z)
• Gradient Descent in Neural Networks as Sequential Learning in RKBS [63.011641517977644]
  We construct an exact power-series representation of the neural network in a finite neighborhood of the initial weights. We prove that, regardless of width, the training sequence produced by gradient descent can be exactly replicated by regularized sequential learning.
  arXiv  Detail & Related papers  (2023-02-01T03:18:07Z)
• Convolutional Neural Generative Coding: Scaling Predictive Coding to Natural Images [79.07468367923619]
  We develop convolutional neural generative coding (Conv-NGC) We implement a flexible neurobiologically-motivated algorithm that progressively refines latent state maps. We study the effectiveness of our brain-inspired neural system on the tasks of reconstruction and image denoising.
  arXiv  Detail & Related papers  (2022-11-22T06:42:41Z)
• NAR-Former: Neural Architecture Representation Learning towards Holistic Attributes Prediction [37.357949900603295]
  We propose a neural architecture representation model that can be used to estimate attributes holistically. Experiment results show that our proposed framework can be used to predict the latency and accuracy attributes of both cell architectures and whole deep neural networks.
  arXiv  Detail & Related papers  (2022-11-15T10:15:21Z)
• Neural Adaptive SCEne Tracing [24.781844909539686]
  We present NAScenT, the first neural rendering method based on directly training a hybrid explicit-implicit neural representation. NAScenT is capable of reconstructing challenging scenes including both large, sparsely populated volumes like UAV captured outdoor environments.
  arXiv  Detail & Related papers  (2022-02-28T10:27:23Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Self-Denoising Neural Networks for Few Shot Learning [66.38505903102373]
  We present a new training scheme that adds noise at multiple stages of an existing neural architecture while simultaneously learning to be robust to this added noise. This architecture, which we call a Self-Denoising Neural Network (SDNN), can be applied easily to most modern convolutional neural architectures.
  arXiv  Detail & Related papers  (2021-10-26T03:28:36Z)
• Feature Alignment for Approximated Reversibility in Neural Networks [0.0]
  We introduce feature alignment, a technique for obtaining approximate reversibility in artificial neural networks. We show that the technique can be modified for training neural networks locally, saving computational memory resources.
  arXiv  Detail & Related papers  (2021-06-23T17:42:47Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.