Unveiling the Potential of Superexpressive Networks in Implicit Neural Representations

• URL: http://arxiv.org/abs/2503.21166v1
• Date: Thu, 27 Mar 2025 05:36:12 GMT
• Title: Unveiling the Potential of Superexpressive Networks in Implicit Neural Representations
• Authors: Uvini Balasuriya Mudiyanselage, Woojin Cho, Minju Jo, Noseong Park, Kookjin Lee,
• Abstract summary: Superexpressive networks employ a specialized network structure characterized by having an additional dimension, namely width, depth, and height''<n>We show that superexpressive networks can surpass recent implicit neural representations that use highly-specialized nonlinear activation functions.
• Score: 23.561956415242584
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this study, we examine the potential of one of the ``superexpressive'' networks in the context of learning neural functions for representing complex signals and performing machine learning downstream tasks. Our focus is on evaluating their performance on computer vision and scientific machine learning tasks including signal representation/inverse problems and solutions of partial differential equations. Through an empirical investigation in various benchmark tasks, we demonstrate that superexpressive networks, as proposed by [Zhang et al. NeurIPS, 2022], which employ a specialized network structure characterized by having an additional dimension, namely width, depth, and ``height'', can surpass recent implicit neural representations that use highly-specialized nonlinear activation functions.

Related papers

• Identifying Sub-networks in Neural Networks via Functionally Similar Representations [41.028797971427124]
  We take a step toward automating the understanding of the network by investigating the existence of distinct sub-networks.<n>Specifically, we explore a novel automated and task-agnostic approach based on the notion of functionally similar representations within neural networks.<n>We show the proposed approach offers meaningful insights into the behavior of neural networks with minimal human and computational cost.
  arXiv  Detail & Related papers  (2024-10-21T20:19:00Z)
• Coding schemes in neural networks learning classification tasks [52.22978725954347]
  We investigate fully-connected, wide neural networks learning classification tasks. We show that the networks acquire strong, data-dependent features. Surprisingly, the nature of the internal representations depends crucially on the neuronal nonlinearity.
  arXiv  Detail & Related papers  (2024-06-24T14:50:05Z)
• Towards Scalable and Versatile Weight Space Learning [51.78426981947659]
  This paper introduces the SANE approach to weight-space learning. Our method extends the idea of hyper-representations towards sequential processing of subsets of neural network weights.
  arXiv  Detail & Related papers  (2024-06-14T13:12:07Z)
• Mechanistic Neural Networks for Scientific Machine Learning [58.99592521721158]
  We present Mechanistic Neural Networks, a neural network design for machine learning applications in the sciences. It incorporates a new Mechanistic Block in standard architectures to explicitly learn governing differential equations as representations. Central to our approach is a novel Relaxed Linear Programming solver (NeuRLP) inspired by a technique that reduces solving linear ODEs to solving linear programs.
  arXiv  Detail & Related papers  (2024-02-20T15:23:24Z)
• Fourier Circuits in Neural Networks and Transformers: A Case Study of Modular Arithmetic with Multiple Inputs [35.212818841550835]
  One-hidden layer neural networks and one-layer Transformers are studied.<n>One-hidden layer neural networks attain a maximum $ L_2,k+1 $-margin on a dataset.<n>We observe similar computational mechanisms in attention of one-layer Transformers.
  arXiv  Detail & Related papers  (2024-02-12T05:52:06Z)
• Feature emergence via margin maximization: case studies in algebraic tasks [4.401622714202886]
  We show that trained neural networks employ features corresponding to irreducible group-theoretic representations to perform compositions in general groups. More generally, we hope our techniques can help to foster a deeper understanding of why neural networks adopt specific computational strategies.
  arXiv  Detail & Related papers  (2023-11-13T18:56:33Z)
• The Multiple Subnetwork Hypothesis: Enabling Multidomain Learning by Isolating Task-Specific Subnetworks in Feedforward Neural Networks [0.0]
  We identify a methodology and network representational structure which allows a pruned network to employ previously unused weights to learn subsequent tasks. We show that networks trained using our approaches are able to learn multiple tasks, which may be related or unrelated, in parallel or in sequence without sacrificing performance on any task or exhibiting catastrophic forgetting.
  arXiv  Detail & Related papers  (2022-07-18T15:07:13Z)
• A survey on recently proposed activation functions for Deep Learning [0.0]
  This survey discusses the main concepts of activation functions in neural networks. It includes a brief introduction to deep neural networks, a summary of what are activation functions and how they are used in neural networks, their most common properties, the different types of activation functions, some of the challenges, limitations, and alternative solutions faced by activation functions.
  arXiv  Detail & Related papers  (2022-04-06T16:21:52Z)
• Quasi-orthogonality and intrinsic dimensions as measures of learning and generalisation [55.80128181112308]
  We show that dimensionality and quasi-orthogonality of neural networks' feature space may jointly serve as network's performance discriminants. Our findings suggest important relationships between the networks' final performance and properties of their randomly initialised feature spaces.
  arXiv  Detail & Related papers  (2022-03-30T21:47:32Z)
• What can linearized neural networks actually say about generalization? [67.83999394554621]
  In certain infinitely-wide neural networks, the neural tangent kernel (NTK) theory fully characterizes generalization. We show that the linear approximations can indeed rank the learning complexity of certain tasks for neural networks. Our work provides concrete examples of novel deep learning phenomena which can inspire future theoretical research.
  arXiv  Detail & Related papers  (2021-06-12T13:05:11Z)
• A neural anisotropic view of underspecification in deep learning [60.119023683371736]
  We show that the way neural networks handle the underspecification of problems is highly dependent on the data representation. Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.
  arXiv  Detail & Related papers  (2021-04-29T14:31:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.