Structured Knowledge Accumulation: An Autonomous Framework for Layer-Wise Entropy Reduction in Neural Learning

• URL: http://arxiv.org/abs/2503.13942v1
• Date: Tue, 18 Mar 2025 06:14:20 GMT
• Title: Structured Knowledge Accumulation: An Autonomous Framework for Layer-Wise Entropy Reduction in Neural Learning
• Authors: Bouarfa Mahi Quantiota,
• Abstract summary: We introduce the Structured Knowledge Accumulation (SKA) framework, which reinterprets entropy as a dynamic, layer-wise measure of knowledge alignment in neural networks.<n>SKA defines entropy in terms of knowledge vectors and their influence on decision probabilities across multiple layers.<n>This approach provides a scalable, biologically plausible alternative to gradient-based learning, bridging information theory and artificial intelligence.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce the Structured Knowledge Accumulation (SKA) framework, which reinterprets entropy as a dynamic, layer-wise measure of knowledge alignment in neural networks. Instead of relying on traditional gradient-based optimization, SKA defines entropy in terms of knowledge vectors and their influence on decision probabilities across multiple layers. This formulation naturally leads to the emergence of activation functions such as the sigmoid as a consequence of entropy minimization. Unlike conventional backpropagation, SKA allows each layer to optimize independently by aligning its knowledge representation with changes in decision probabilities. As a result, total network entropy decreases in a hierarchical manner, allowing knowledge structures to evolve progressively. This approach provides a scalable, biologically plausible alternative to gradient-based learning, bridging information theory and artificial intelligence while offering promising applications in resource-constrained and parallel computing environments.

Related papers

• Structured Knowledge Accumulation: The Principle of Entropic Least Action in Forward-Only Neural Learning [0.0]
  We introduce two core concepts: the Net function and the characteristic time property of neural learning. By understanding learning as a time-based process, we open new directions for building efficient, robust, and biologically-inspired AI systems.
  arXiv  Detail & Related papers  (2025-04-04T07:00:27Z)
• Structural Entropy Guided Probabilistic Coding [52.01765333755793]
  We propose a novel structural entropy-guided probabilistic coding model, named SEPC. We incorporate the relationship between latent variables into the optimization by proposing a structural entropy regularization loss. Experimental results across 12 natural language understanding tasks, including both classification and regression tasks, demonstrate the superior performance of SEPC.
  arXiv  Detail & Related papers  (2024-12-12T00:37:53Z)
• Effective Reinforcement Learning Based on Structural Information Principles [19.82391136775341]
  We propose a novel and general Structural Information principles-based framework for effective Decision-Making, namely SIDM. SIDM can be flexibly incorporated into various single-agent and multi-agent RL algorithms, enhancing their performance.
  arXiv  Detail & Related papers  (2024-04-15T13:02:00Z)
• Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation [49.44309457870649]
  We present Layer-wise Feedback Propagation (LFP), a novel training principle for neural network-like predictors. LFP decomposes a reward to individual neurons based on their respective contributions to solving a given task. Our method then implements a greedy approach reinforcing helpful parts of the network and weakening harmful ones.
  arXiv  Detail & Related papers  (2023-08-23T10:48:28Z)
• Reparameterization through Spatial Gradient Scaling [69.27487006953852]
  Reparameterization aims to improve the generalization of deep neural networks by transforming convolutional layers into equivalent multi-branched structures during training. We present a novel spatial gradient scaling method to redistribute learning focus among weights in convolutional networks.
  arXiv  Detail & Related papers  (2023-03-05T17:57:33Z)
• ConCerNet: A Contrastive Learning Based Framework for Automated Conservation Law Discovery and Trustworthy Dynamical System Prediction [82.81767856234956]
  This paper proposes a new learning framework named ConCerNet to improve the trustworthiness of the DNN based dynamics modeling. We show that our method consistently outperforms the baseline neural networks in both coordinate error and conservation metrics.
  arXiv  Detail & Related papers  (2023-02-11T21:07:30Z)
• The Mori-Zwanzig formulation of deep learning [3.2851683371946754]
  We develop a new formulation of deep learning based on the Mori-Zwanzig formalism of irreversible statistical mechanics. New equations can be used as a starting point to develop new effective parameterizations of deep neural networks.
  arXiv  Detail & Related papers  (2022-09-12T18:44:50Z)
• Knowledge Enhanced Neural Networks for relational domains [83.9217787335878]
  We focus on a specific method, KENN, a Neural-Symbolic architecture that injects prior logical knowledge into a neural network. In this paper, we propose an extension of KENN for relational data.
  arXiv  Detail & Related papers  (2022-05-31T13:00:34Z)
• Towards the One Learning Algorithm Hypothesis: A System-theoretic Approach [0.0]
  The existence of a universal learning architecture in human cognition is a widely spread conjecture supported by experimental findings from neuroscience. We develop a closed-loop system with three main components: (i) a multi-resolution analysis pre-processor, (ii) a group-invariant feature extractor, and (iii) a progressive knowledge-based learning module. We introduce a novel learning algorithm that constructs progressively growing knowledge representations in multiple resolutions.
  arXiv  Detail & Related papers  (2021-12-04T05:54:33Z)
• Clustering-Based Interpretation of Deep ReLU Network [17.234442722611803]
  We recognize that the non-linear behavior of the ReLU function gives rise to a natural clustering. We propose a method to increase the level of interpretability of a fully connected feedforward ReLU neural network.
  arXiv  Detail & Related papers  (2021-10-13T09:24:11Z)
• Influence Estimation and Maximization via Neural Mean-Field Dynamics [60.91291234832546]
  We propose a novel learning framework using neural mean-field (NMF) dynamics for inference and estimation problems. Our framework can simultaneously learn the structure of the diffusion network and the evolution of node infection probabilities.
  arXiv  Detail & Related papers  (2021-06-03T00:02:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.