Architecture of Information

• URL: http://arxiv.org/abs/2503.21794v1
• Date: Fri, 21 Mar 2025 14:48:41 GMT
• Title: Architecture of Information
• Authors: Yurii Parzhyn,
• Abstract summary: The paper explores an approach to constructing energy landscapes of a formal neuron and multilayer artificial neural networks (ANNs)<n>The study of informational and thermodynamic entropy in formal neuron and ANN models leads to the conclusion about the energetic nature of informational entropy.<n>The presented research makes it possible to formulate a formal definition of information in terms of the interaction processes between the internal and external energy of the system.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The paper explores an approach to constructing energy landscapes of a formal neuron and multilayer artificial neural networks (ANNs). Their analysis makes it possible to determine the conceptual limitations of both classification ANNs (e.g., MLP or CNN) and generative ANN models. The study of informational and thermodynamic entropy in formal neuron and ANN models leads to the conclusion about the energetic nature of informational entropy. The application of the Gibbs free energy concept allows representing the output information of ANNs as the structured part of enthalpy. Modeling ANNs as energy systems makes it possible to interpret the structure of their internal energy as an internal model of the external world, which self-organizes based on the interaction of the system's internal energy components. The control of the self-organization and evolution process of this model is carried out through an energy function (analogous to the Lyapunov function) based on reduction operators. This makes it possible to introduce a new approach to constructing self-organizing and evolutionary ANNs with direct learning, which does not require additional external algorithms. The presented research makes it possible to formulate a formal definition of information in terms of the interaction processes between the internal and external energy of the system.

Related papers

• Thermodynamic Bound on Energy and Negentropy Costs of Inference in Deep Neural Networks [0.0]
  The fundamental thermodynamic bound is derived for the energy cost of inference in Deep Neural Networks (DNNs)<n>We show that the linear operations in DNNs can, in principle, be performed reversibly, whereas the non-linear activation functions impose an unavoidable energy cost.
  arXiv  Detail & Related papers  (2025-03-13T02:35:07Z)
• Neural Message Passing Induced by Energy-Constrained Diffusion [79.9193447649011]
  We propose an energy-constrained diffusion model as a principled interpretable framework for understanding the mechanism of MPNNs. We show that the new model can yield promising performance for cases where the data structures are observed (as a graph), partially observed or completely unobserved.
  arXiv  Detail & Related papers  (2024-09-13T17:54:41Z)
• Enhancing learning in spiking neural networks through neuronal heterogeneity and neuromodulatory signaling [52.06722364186432]
  We propose a biologically-informed framework for enhancing artificial neural networks (ANNs) Our proposed dual-framework approach highlights the potential of spiking neural networks (SNNs) for emulating diverse spiking behaviors. We outline how the proposed approach integrates brain-inspired compartmental models and task-driven SNNs, bioinspiration and complexity.
  arXiv  Detail & Related papers  (2024-07-05T14:11:28Z)
• Theory and implementation of inelastic Constitutive Artificial Neural Networks [0.0]
  We extend the Constitutive Artificial Neural Networks (CANNs) to inelastic materials (iCANN) We demonstrate that the iCANN is capable of autonomously discovering models for artificially generated data. Our vision is that the iCANN will reveal to us new ways to find the various inelastic phenomena hidden in the data and to understand their interaction.
  arXiv  Detail & Related papers  (2023-11-10T20:13:29Z)
• Unraveling Feature Extraction Mechanisms in Neural Networks [10.13842157577026]
  We propose a theoretical approach based on Neural Tangent Kernels (NTKs) to investigate such mechanisms. We reveal how these models leverage statistical features during gradient descent and how they are integrated into final decisions. We find that while self-attention and CNN models may exhibit limitations in learning n-grams, multiplication-based models seem to excel in this area.
  arXiv  Detail & Related papers  (2023-10-25T04:22:40Z)
• Inferring Relational Potentials in Interacting Systems [56.498417950856904]
  We propose Neural Interaction Inference with Potentials (NIIP) as an alternative approach to discover such interactions. NIIP assigns low energy to the subset of trajectories which respect the relational constraints observed. It allows trajectory manipulation, such as interchanging interaction types across separately trained models, as well as trajectory forecasting.
  arXiv  Detail & Related papers  (2023-10-23T00:44:17Z)
• A Neuro-mimetic Realization of the Common Model of Cognition via Hebbian Learning and Free Energy Minimization [55.11642177631929]
  Large neural generative models are capable of synthesizing semantically rich passages of text or producing complex images. We discuss the COGnitive Neural GENerative system, such an architecture that casts the Common Model of Cognition.
  arXiv  Detail & Related papers  (2023-10-14T23:28:48Z)
• Energy Transformer [64.22957136952725]
  Our work combines aspects of three promising paradigms in machine learning, namely, attention mechanism, energy-based models, and associative memory. We propose a novel architecture, called the Energy Transformer (or ET for short), that uses a sequence of attention layers that are purposely designed to minimize a specifically engineered energy function.
  arXiv  Detail & Related papers  (2023-02-14T18:51:22Z)
• Learning Energy-Based Approximate Inference Networks for Structured Applications in NLP [8.426855646402238]
  The dissertation begins with a general introduction to energy-based models. We propose a method in which we train a neural network to do argmax inference under a structured energy function. We then develop ways of jointly learning energy functions and inference networks using an adversarial learning framework.
  arXiv  Detail & Related papers  (2021-08-27T22:48:20Z)
• Continuous Learning and Adaptation with Membrane Potential and Activation Threshold Homeostasis [91.3755431537592]
  This paper presents the Membrane Potential and Activation Threshold Homeostasis (MPATH) neuron model. The model allows neurons to maintain a form of dynamic equilibrium by automatically regulating their activity when presented with input. Experiments demonstrate the model's ability to adapt to and continually learn from its input.
  arXiv  Detail & Related papers  (2021-04-22T04:01:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.