From Activation to Initialization: Scaling Insights for Optimizing Neural Fields

• URL: http://arxiv.org/abs/2403.19205v1
• Date: Thu, 28 Mar 2024 08:06:48 GMT
• Title: From Activation to Initialization: Scaling Insights for Optimizing Neural Fields
• Authors: Hemanth Saratchandran, Sameera Ramasinghe, Simon Lucey,
• Abstract summary: This article aims to address the gap by delving into the interplay between initialization and activation, providing a foundational basis for the robust optimization of Neural Fields. Our theoretical insights reveal a deep-seated connection among network initialization, architectural choices, and the optimization process, emphasizing the need for a holistic approach when designing cutting-edge Neural Fields.
• Score: 37.52425975916322
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the realm of computer vision, Neural Fields have gained prominence as a contemporary tool harnessing neural networks for signal representation. Despite the remarkable progress in adapting these networks to solve a variety of problems, the field still lacks a comprehensive theoretical framework. This article aims to address this gap by delving into the intricate interplay between initialization and activation, providing a foundational basis for the robust optimization of Neural Fields. Our theoretical insights reveal a deep-seated connection among network initialization, architectural choices, and the optimization process, emphasizing the need for a holistic approach when designing cutting-edge Neural Fields.

Related papers

• Differentiable Convex Optimization Layers in Neural Architectures: Foundations and Perspectives [0.0]
  The integration of optimization problems within neural network architectures represents a shift from traditional approaches to handling constraints in deep learning. A recent advance in this field has enabled the direct embedding of optimization layers as differentiable components within deep networks. This work synthesizes developments at the intersection of optimization theory and deep learning, offering insights into both current capabilities and future research directions.
  arXiv  Detail & Related papers  (2024-12-30T03:18:24Z)
• Evidence, Definitions and Algorithms regarding the Existence of Cohesive-Convergence Groups in Neural Network Optimization [0.0]
  Understanding the process of neural networks is one of the most complex and crucial issues in the field of machine learning. This paper focuses on the theoretical convergence of artificial neural networks.
  arXiv  Detail & Related papers  (2024-03-08T13:23:42Z)
• Topology Optimization using Neural Networks with Conditioning Field Initialization for Improved Efficiency [2.575019247624295]
  We show that by using a prior initial field on the unoptimized domain, the efficiency of neural network based topology optimization can be further improved. We employ the strain energy field calculated on the initial design domain as an additional conditioning field input to the neural network throughout the optimization.
  arXiv  Detail & Related papers  (2023-05-17T07:42:24Z)
• Optimisation & Generalisation in Networks of Neurons [8.078758339149822]
  The goal of this thesis is to develop the optimisation and generalisation theoretic foundations of learning in artificial neural networks. A new theoretical framework is proposed for deriving architecture-dependent first-order optimisation algorithms. A new correspondence is proposed between ensembles of networks and individual networks.
  arXiv  Detail & Related papers  (2022-10-18T18:58:40Z)
• Learning Structures for Deep Neural Networks [99.8331363309895]
  We propose to adopt the efficient coding principle, rooted in information theory and developed in computational neuroscience. We show that sparse coding can effectively maximize the entropy of the output signals. Our experiments on a public image classification dataset demonstrate that using the structure learned from scratch by our proposed algorithm, one can achieve a classification accuracy comparable to the best expert-designed structure.
  arXiv  Detail & Related papers  (2021-05-27T12:27:24Z)
• Developing Constrained Neural Units Over Time [81.19349325749037]
  This paper focuses on an alternative way of defining Neural Networks, that is different from the majority of existing approaches. The structure of the neural architecture is defined by means of a special class of constraints that are extended also to the interaction with data. The proposed theory is cast into the time domain, in which data are presented to the network in an ordered manner.
  arXiv  Detail & Related papers  (2020-09-01T09:07:25Z)
• Learning Connectivity of Neural Networks from a Topological Perspective [80.35103711638548]
  We propose a topological perspective to represent a network into a complete graph for analysis. By assigning learnable parameters to the edges which reflect the magnitude of connections, the learning process can be performed in a differentiable manner. This learning process is compatible with existing networks and owns adaptability to larger search spaces and different tasks.
  arXiv  Detail & Related papers  (2020-08-19T04:53:31Z)
• Generalization bound of globally optimal non-convex neural network training: Transportation map estimation by infinite dimensional Langevin dynamics [50.83356836818667]
  We introduce a new theoretical framework to analyze deep learning optimization with connection to its generalization error. Existing frameworks such as mean field theory and neural tangent kernel theory for neural network optimization analysis typically require taking limit of infinite width of the network to show its global convergence.
  arXiv  Detail & Related papers  (2020-07-11T18:19:50Z)
• Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks [54.27962244835622]
  This paper proposes a new mean-field framework for over- parameterized deep neural networks (DNNs) In this framework, a DNN is represented by probability measures and functions over its features in the continuous limit. We illustrate the framework via the standard DNN and the Residual Network (Res-Net) architectures.
  arXiv  Detail & Related papers  (2020-07-03T01:37:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.