Physics-inspired Energy Transition Neural Network for Sequence Learning

• URL: http://arxiv.org/abs/2505.03281v1
• Date: Tue, 06 May 2025 08:07:15 GMT
• Title: Physics-inspired Energy Transition Neural Network for Sequence Learning
• Authors: Zhou Wu, Junyi An, Baile Xu, Furao Shen, Jian Zhao,
• Abstract summary: In this study, we explore the capabilities of pure RNNs and reassess their long-term learning mechanisms.<n>Inspired by the physics energy transition models that track energy changes over time, we propose a effective recurrent structure called thePhysics-inspired Energy Transition Neural Network" (PETNN)<n>Our study presents an optimal foundational recurrent architecture and highlights the potential for developing effective recurrent neural networks in fields currently dominated by Transformer.
• Score: 14.111325019623596
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, the superior performance of Transformers has made them a more robust and scalable solution for sequence modeling than traditional recurrent neural networks (RNNs). However, the effectiveness of Transformer in capturing long-term dependencies is primarily attributed to their comprehensive pair-modeling process rather than inherent inductive biases toward sequence semantics. In this study, we explore the capabilities of pure RNNs and reassess their long-term learning mechanisms. Inspired by the physics energy transition models that track energy changes over time, we propose a effective recurrent structure called the``Physics-inspired Energy Transition Neural Network" (PETNN). We demonstrate that PETNN's memory mechanism effectively stores information over long-term dependencies. Experimental results indicate that PETNN outperforms transformer-based methods across various sequence tasks. Furthermore, owing to its recurrent nature, PETNN exhibits significantly lower complexity. Our study presents an optimal foundational recurrent architecture and highlights the potential for developing effective recurrent neural networks in fields currently dominated by Transformer.

Related papers

• Fractional Spike Differential Equations Neural Network with Efficient Adjoint Parameters Training [63.3991315762955]
  Spiking Neural Networks (SNNs) draw inspiration from biological neurons to create realistic models for brain-like computation.<n>Most existing SNNs assume a single time constant for neuronal membrane voltage dynamics, modeled by first-order ordinary differential equations (ODEs) with Markovian characteristics.<n>We propose the Fractional SPIKE Differential Equation neural network (fspikeDE), which captures long-term dependencies in membrane voltage and spike trains through fractional-order dynamics.
  arXiv  Detail & Related papers  (2025-07-22T18:20:56Z)
• Langevin Flows for Modeling Neural Latent Dynamics [81.81271685018284]
  We introduce LangevinFlow, a sequential Variational Auto-Encoder where the time evolution of latent variables is governed by the underdamped Langevin equation.<n>Our approach incorporates physical priors -- such as inertia, damping, a learned potential function, and forces -- to represent both autonomous and non-autonomous processes in neural systems.<n>Our method outperforms state-of-the-art baselines on synthetic neural populations generated by a Lorenz attractor.
  arXiv  Detail & Related papers  (2025-07-15T17:57:48Z)
• Energy-Efficient Deep Reinforcement Learning with Spiking Transformers [3.037387520023979]
  Spiking neural networks (SNNs) offer an energy-efficient alternative for machine learning.<n>New Spike-Transformer Reinforcement Learning (STRL) algorithm combines the energy efficiency of SNNs with the powerful decision-making capabilities of reinforcement learning.<n>SNN Transformer achieves significantly improved policy performance compared to conventional agent-based Transformers.
  arXiv  Detail & Related papers  (2025-05-20T15:52:43Z)
• Long-term excitation energy transfer predicted by a modified convolutional neural networks in the FMO complexes [0.49157446832511503]
  We propose an efficient CNNs scheme incorporating novel redundant time-functions to predict 100 picosecond (ps) excitation energy transfer (EET) in Fenna-Matthews-Olson complexes.<n>This method simplifies optimization and enhances learning efficiency, and demonstrates the accuracy, robustness, and efficiency of our approach in predicting quantum dissipative dynamics.
  arXiv  Detail & Related papers  (2025-03-21T12:40:39Z)
• Temporal Misalignment in ANN-SNN Conversion and Its Mitigation via Probabilistic Spiking Neurons [17.73940693302129]
  Spiking Neural Networks (SNNs) offer a more energy-efficient alternative to Artificial Neural Networks (ANNs)<n>In this work, we identify a phenomenon in the ANN-SNN conversion framework, termed temporal misalignment.<n>We introduce biologically plausible two-phase probabilistic (TPP) spiking neurons, further enhancing the conversion process.
  arXiv  Detail & Related papers  (2025-02-20T12:09:30Z)
• Deep-Unrolling Multidimensional Harmonic Retrieval Algorithms on Neuromorphic Hardware [78.17783007774295]
  This paper explores the potential of conversion-based neuromorphic algorithms for highly accurate and energy-efficient single-snapshot multidimensional harmonic retrieval.<n>A novel method for converting the complex-valued convolutional layers and activations into spiking neural networks (SNNs) is developed.<n>The converted SNNs achieve almost five-fold power efficiency at moderate performance loss compared to the original CNNs.
  arXiv  Detail & Related papers  (2024-12-05T09:41:33Z)
• Scalable Mechanistic Neural Networks for Differential Equations and Machine Learning [52.28945097811129]
  We propose an enhanced neural network framework designed for scientific machine learning applications involving long temporal sequences.<n>We reduce the computational time and space complexities from cubic and quadratic with respect to the sequence length, respectively, to linear.<n>Extensive experiments demonstrate that S-MNN matches the original MNN in precision while substantially reducing computational resources.
  arXiv  Detail & Related papers  (2024-10-08T14:27:28Z)
• Contrastive Learning in Memristor-based Neuromorphic Systems [55.11642177631929]
  Spiking neural networks have become an important family of neuron-based models that sidestep many of the key limitations facing modern-day backpropagation-trained deep networks. In this work, we design and investigate a proof-of-concept instantiation of contrastive-signal-dependent plasticity (CSDP), a neuromorphic form of forward-forward-based, backpropagation-free learning.
  arXiv  Detail & Related papers  (2024-09-17T04:48:45Z)
• Exploiting Heterogeneity in Timescales for Sparse Recurrent Spiking Neural Networks for Energy-Efficient Edge Computing [16.60622265961373]
  Spiking Neural Networks (SNNs) represent the forefront of neuromorphic computing. This paper weaves together three groundbreaking studies that revolutionize SNN performance.
  arXiv  Detail & Related papers  (2024-07-08T23:33:12Z)
• Single Neuromorphic Memristor closely Emulates Multiple Synaptic Mechanisms for Energy Efficient Neural Networks [71.79257685917058]
  We demonstrate memristive nano-devices based on SrTiO3 that inherently emulate all these synaptic functions. These memristors operate in a non-filamentary, low conductance regime, which enables stable and energy efficient operation.
  arXiv  Detail & Related papers  (2024-02-26T15:01:54Z)
• Volume-Preserving Transformers for Learning Time Series Data with Structure [0.0]
  We develop a transformer-inspired neural network and use it to learn a dynamical system. We change the activation function of the attention layer to imbue the transformer with structure-preserving properties. This is shown to be of great advantage when applying the neural network to learning the trajectory of a rigid body.
  arXiv  Detail & Related papers  (2023-12-18T13:09:55Z)
• ConCerNet: A Contrastive Learning Based Framework for Automated Conservation Law Discovery and Trustworthy Dynamical System Prediction [82.81767856234956]
  This paper proposes a new learning framework named ConCerNet to improve the trustworthiness of the DNN based dynamics modeling. We show that our method consistently outperforms the baseline neural networks in both coordinate error and conservation metrics.
  arXiv  Detail & Related papers  (2023-02-11T21:07:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.