GmNet: Revisiting Gating Mechanisms From A Frequency View

• URL: http://arxiv.org/abs/2503.22841v1
• Date: Fri, 28 Mar 2025 19:26:45 GMT
• Title: GmNet: Revisiting Gating Mechanisms From A Frequency View
• Authors: Yifan Wang, Xu Ma, Yitian Zhang, Zhongruo Wang, Sung-Cheol Kim, Vahid Mirjalili, Vidya Renganathan, Yun Fu,
• Abstract summary: We study the effect of gating mechanisms on the training dynamics of neural networks from a frequency perspective.<n>We propose a lightweight model designed to efficiently utilize the information of various frequency components.<n>GmNet achieves impressive performance in terms of both effectiveness and efficiency in the image classification task.
• Score: 38.7855775931718
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Gating mechanisms have emerged as an effective strategy integrated into model designs beyond recurrent neural networks for addressing long-range dependency problems. In a broad understanding, it provides adaptive control over the information flow while maintaining computational efficiency. However, there is a lack of theoretical analysis on how the gating mechanism works in neural networks. In this paper, inspired by the {convolution theorem}, we systematically explore the effect of gating mechanisms on the training dynamics of neural networks from a frequency perspective. We investigate the interact between the element-wise product and activation functions in managing the responses to different frequency components. Leveraging these insights, we propose a Gating Mechanism Network (GmNet), a lightweight model designed to efficiently utilize the information of various frequency components. It minimizes the low-frequency bias present in existing lightweight models. GmNet achieves impressive performance in terms of both effectiveness and efficiency in the image classification task.

Related papers

• Exploring Superposition and Interference in State-of-the-Art Low-Parameter Vision Models [0.0]
  We address interference in feature maps, a phenomenon associated with superposition, where neurons simultaneously encode multiple characteristics.<n>Our research suggests that limiting interference can enhance scaling and accuracy in very low-scaled networks (under 1.5M parameters)<n>We propose a proof-of-concept architecture named NoDepth Bottleneck built on mechanistic insights from our experiments, demonstrating robust scaling accuracy on the ImageNet dataset.
  arXiv  Detail & Related papers  (2025-07-21T16:57:25Z)
• Beyond Scaling Curves: Internal Dynamics of Neural Networks Through the NTK Lens [0.5745241788717261]
  We empirically analyze how neural networks behave under data and model scaling through the lens of the neural tangent kernel (NTK)<n>Our findings of standard vision tasks show that similar performance scaling exponents can occur even though the internal model dynamics show opposite behavior.<n>We also address a previously unresolved issue in neural scaling: how convergence to the infinite-width limit affects scaling behavior in finite-width models.
  arXiv  Detail & Related papers  (2025-07-07T14:17:44Z)
• Allostatic Control of Persistent States in Spiking Neural Networks for perception and computation [79.16635054977068]
  We introduce a novel model for updating perceptual beliefs about the environment by extending the concept of Allostasis to the control of internal representations.<n>In this paper, we focus on an application in numerical cognition, where a bump of activity in an attractor network is used as a spatial numerical representation.
  arXiv  Detail & Related papers  (2025-03-20T12:28:08Z)
• Task-Oriented Real-time Visual Inference for IoVT Systems: A Co-design Framework of Neural Networks and Edge Deployment [61.20689382879937]
  Task-oriented edge computing addresses this by shifting data analysis to the edge. Existing methods struggle to balance high model performance with low resource consumption. We propose a novel co-design framework to optimize neural network architecture.
  arXiv  Detail & Related papers  (2024-10-29T19:02:54Z)
• Harnessing Neural Unit Dynamics for Effective and Scalable Class-Incremental Learning [38.09011520275557]
  Class-incremental learning (CIL) aims to train a model to learn new classes from non-stationary data streams without forgetting old ones. We propose a new kind of connectionist model by tailoring neural unit dynamics that adapt the behavior of neural networks for CIL.
  arXiv  Detail & Related papers  (2024-06-04T15:47:03Z)
• Graph Neural Networks for Learning Equivariant Representations of Neural Networks [55.04145324152541]
  We propose to represent neural networks as computational graphs of parameters. Our approach enables a single model to encode neural computational graphs with diverse architectures. We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations.
  arXiv  Detail & Related papers  (2024-03-18T18:01:01Z)
• Mechanistic Neural Networks for Scientific Machine Learning [58.99592521721158]
  We present Mechanistic Neural Networks, a neural network design for machine learning applications in the sciences. It incorporates a new Mechanistic Block in standard architectures to explicitly learn governing differential equations as representations. Central to our approach is a novel Relaxed Linear Programming solver (NeuRLP) inspired by a technique that reduces solving linear ODEs to solving linear programs.
  arXiv  Detail & Related papers  (2024-02-20T15:23:24Z)
• How neural networks learn to classify chaotic time series [77.34726150561087]
  We study the inner workings of neural networks trained to classify regular-versus-chaotic time series. We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
  arXiv  Detail & Related papers  (2023-06-04T08:53:27Z)
• Dynamic Community Detection via Adversarial Temporal Graph Representation Learning [17.487265170798974]
  In this work, an adversarial temporal graph representation learning framework is proposed to detect dynamic communities from a small sample of brain network data. In addition, the framework employs adversarial training to guide the learning of temporal graph representation and optimize the measurable modularity loss to maximize the modularity of community.
  arXiv  Detail & Related papers  (2022-06-29T08:44:22Z)
• On the role of feedback in visual processing: a predictive coding perspective [0.6193838300896449]
  We consider deep convolutional networks (CNNs) as models of feed-forward visual processing and implement Predictive Coding (PC) dynamics. We find that the network increasingly relies on top-down predictions as the noise level increases. In addition, the accuracy of the network implementing PC dynamics significantly increases over time-steps, compared to its equivalent forward network.
  arXiv  Detail & Related papers  (2021-06-08T10:07:23Z)
• Machine Learning Link Inference of Noisy Delay-coupled Networks with Opto-Electronic Experimental Tests [1.0766846340954257]
  We devise a machine learning technique to solve the general problem of inferring network links that have time-delays. We first train a type of machine learning system known as reservoir computing to mimic the dynamics of the unknown network. We formulate and test a technique that uses the trained parameters of the reservoir system output layer to deduce an estimate of the unknown network structure.
  arXiv  Detail & Related papers  (2020-10-29T00:24:13Z)
• A Principle of Least Action for the Training of Neural Networks [10.342408668490975]
  We show the presence of a low kinetic energy displacement bias in the transport map of the network, and link this bias with generalization performance. We propose a new learning algorithm, which automatically adapts to the complexity of the given task, and leads to networks with a high generalization ability even in low data regimes.
  arXiv  Detail & Related papers  (2020-09-17T15:37:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.