EMN: Brain-inspired Elastic Memory Network for Quick Domain Adaptive Feature Mapping

• URL: http://arxiv.org/abs/2402.14598v2
• Date: Mon, 17 Mar 2025 08:34:07 GMT
• Title: EMN: Brain-inspired Elastic Memory Network for Quick Domain Adaptive Feature Mapping
• Authors: Jianming Lv, Chengjun Wang, Depin Liang, Qianli Ma, Wei Chen, Xueqi Cheng,
• Abstract summary: We propose a novel gradient-free Elastic Memory Network to support quick fine-tuning of the mapping between features and prediction.<n>EMN adopts randomly connected neurons to memorize the association of features and labels, where the signals in the network are propagated as impulses.<n>EMN can achieve up to 10% enhancement of performance while only needing less than 1% timing cost of traditional domain adaptation methods.
• Score: 57.197694698750404
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Utilizing unlabeled data in the target domain to perform continuous optimization is critical to enhance the generalization ability of neural networks. Most domain adaptation methods focus on time-consuming optimization of deep feature extractors, which limits the deployment on lightweight edge devices. Inspired by the memory mechanism and powerful generalization ability of biological neural networks in human brains, we propose a novel gradient-free Elastic Memory Network, namely EMN, to support quick fine-tuning of the mapping between features and prediction without heavy optimization of deep features. In particular, EMN adopts randomly connected neurons to memorize the association of features and labels, where the signals in the network are propagated as impulses, and the prediction is made by associating the memories stored on neurons based on their confidence. More importantly, EMN supports reinforced memorization of feature mapping based on unlabeled data to quickly adapt to a new domain. Experiments based on four cross-domain real-world datasets show that EMN can achieve up to 10% enhancement of performance while only needing less than 1% timing cost of traditional domain adaptation methods.

Related papers

• Spiking Meets Attention: Efficient Remote Sensing Image Super-Resolution with Attention Spiking Neural Networks [57.17129753411926]
  Spiking neural networks (SNNs) are emerging as a promising alternative to traditional artificial neural networks (ANNs) We propose SpikeSR, which achieves state-of-the-art performance across various remote sensing benchmarks such as AID, DOTA, and DIOR.
  arXiv  Detail & Related papers  (2025-03-06T09:06:06Z)
• Transported Memory Networks accelerating Computational Fluid Dynamics [0.0699049312989311]
  Transported Memory Networks is a novel architecture that draws inspiration from both traditional turbulence models and recurrent neural networks. Our results show that it is point-wise and statistically comparable to, or improves upon, previous methods in terms of both accuracy and computational efficiency.
  arXiv  Detail & Related papers  (2025-02-25T19:12:48Z)
• Activation-wise Propagation: A Universal Strategy to Break Timestep Constraints in Spiking Neural Networks for 3D Data Processing [29.279985043923386]
  We introduce Activation-wise Membrane Potential Propagation (AMP2), a novel state update mechanism for spiking neurons. Inspired by skip connections in deep networks, AMP2 incorporates the membrane potential of neurons into network, eliminating the need for iterative updates. Our method achieves significant improvements across various 3D modalities, including 3D point clouds and event streams.
  arXiv  Detail & Related papers  (2025-02-18T11:52:25Z)
• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• Towards Interpretable Deep Local Learning with Successive Gradient Reconciliation [70.43845294145714]
  Relieving the reliance of neural network training on a global back-propagation (BP) has emerged as a notable research topic. We propose a local training strategy that successively regularizes the gradient reconciliation between neighboring modules. Our method can be integrated into both local-BP and BP-free settings.
  arXiv  Detail & Related papers  (2024-06-07T19:10:31Z)
• Heterogenous Memory Augmented Neural Networks [84.29338268789684]
  We introduce a novel heterogeneous memory augmentation approach for neural networks. By introducing learnable memory tokens with attention mechanism, we can effectively boost performance without huge computational overhead. We show our approach on various image and graph-based tasks under both in-distribution (ID) and out-of-distribution (OOD) conditions.
  arXiv  Detail & Related papers  (2023-10-17T01:05:28Z)
• Sharing Leaky-Integrate-and-Fire Neurons for Memory-Efficient Spiking Neural Networks [9.585985556876537]
  Non-linear activation of Leaky-Integrate-and-Fire (LIF) neurons requires additional memory to store a membrane voltage to capture the temporal dynamics of spikes. We propose a simple and effective solution, EfficientLIF-Net, which shares the LIF neurons across different layers and channels. Our EfficientLIF-Net achieves comparable accuracy with the standard SNNs while bringing up to 4.3X forward memory efficiency and 21.9X backward memory efficiency for LIF neurons.
  arXiv  Detail & Related papers  (2023-05-26T22:55:26Z)
• MF-NeRF: Memory Efficient NeRF with Mixed-Feature Hash Table [62.164549651134465]
  We propose MF-NeRF, a memory-efficient NeRF framework that employs a Mixed-Feature hash table to improve memory efficiency and reduce training time while maintaining reconstruction quality. Our experiments with state-of-the-art Instant-NGP, TensoRF, and DVGO, indicate our MF-NeRF could achieve the fastest training time on the same GPU hardware with similar or even higher reconstruction quality.
  arXiv  Detail & Related papers  (2023-04-25T05:44:50Z)
• Selective Memory Recursive Least Squares: Recast Forgetting into Memory in RBF Neural Network Based Real-Time Learning [2.31120983784623]
  In radial basis function neural network (RBFNN) based real-time learning tasks, forgetting mechanisms are widely used. This paper proposes a real-time training method named selective memory recursive least squares (SMRLS) in which the classical forgetting mechanisms are recast into a memory mechanism. With SMRLS, the input space of the RBFNN is evenly divided into a finite number of partitions and a synthesized objective function is developed using synthesized samples from each partition.
  arXiv  Detail & Related papers  (2022-11-15T05:29:58Z)
• Scaling Forward Gradient With Local Losses [117.22685584919756]
  Forward learning is a biologically plausible alternative to backprop for learning deep neural networks. We show that it is possible to substantially reduce the variance of the forward gradient by applying perturbations to activations rather than weights. Our approach matches backprop on MNIST and CIFAR-10 and significantly outperforms previously proposed backprop-free algorithms on ImageNet.
  arXiv  Detail & Related papers  (2022-10-07T03:52:27Z)
• Learning with Local Gradients at the Edge [14.94491070863641]
  We present a novel backpropagation-free optimization algorithm dubbed Target Projection Gradient Descent (tpSGD) tpSGD generalizes direct random target projection to work with arbitrary loss functions. We evaluate the performance of tpSGD in training deep neural networks and extend the approach to multi-layer RNNs.
  arXiv  Detail & Related papers  (2022-08-17T19:51:06Z)
• Predictive coding, precision and natural gradients [2.1601966913620325]
  We show that hierarchical predictive coding networks with learnable precision are able to solve various supervised and unsupervised learning tasks. When applied to unsupervised auto-encoding of image inputs, the deterministic network produces hierarchically organized and disentangled embeddings.
  arXiv  Detail & Related papers  (2021-11-12T21:05:03Z)
• Stagewise Unsupervised Domain Adaptation with Adversarial Self-Training for Road Segmentation of Remote Sensing Images [93.50240389540252]
  Road segmentation from remote sensing images is a challenging task with wide ranges of application potentials. We propose a novel stagewise domain adaptation model called RoadDA to address the domain shift (DS) issue in this field. Experiment results on two benchmarks demonstrate that RoadDA can efficiently reduce the domain gap and outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-08-28T09:29:14Z)
• Non-Gradient Manifold Neural Network [79.44066256794187]
  Deep neural network (DNN) generally takes thousands of iterations to optimize via gradient descent. We propose a novel manifold neural network based on non-gradient optimization.
  arXiv  Detail & Related papers  (2021-06-15T06:39:13Z)
• Influence Estimation and Maximization via Neural Mean-Field Dynamics [60.91291234832546]
  We propose a novel learning framework using neural mean-field (NMF) dynamics for inference and estimation problems. Our framework can simultaneously learn the structure of the diffusion network and the evolution of node infection probabilities.
  arXiv  Detail & Related papers  (2021-06-03T00:02:05Z)
• Stochastic Markov Gradient Descent and Training Low-Bit Neural Networks [77.34726150561087]
  We introduce Gradient Markov Descent (SMGD), a discrete optimization method applicable to training quantized neural networks. We provide theoretical guarantees of algorithm performance as well as encouraging numerical results.
  arXiv  Detail & Related papers  (2020-08-25T15:48:15Z)
• Randomized Automatic Differentiation [22.95414996614006]
  We develop a general framework and approach for randomized automatic differentiation (RAD) RAD can allow unbiased estimates to be computed with reduced memory in return for variance. We show that RAD converges in fewer iterations than using a small batch size for feedforward networks, and in a similar number for recurrent networks.
  arXiv  Detail & Related papers  (2020-07-20T19:03:44Z)
• Optimizing Memory Placement using Evolutionary Graph Reinforcement Learning [56.83172249278467]
  We introduce Evolutionary Graph Reinforcement Learning (EGRL), a method designed for large search spaces. We train and validate our approach directly on the Intel NNP-I chip for inference. We additionally achieve 28-78% speed-up compared to the native NNP-I compiler on all three workloads.
  arXiv  Detail & Related papers  (2020-07-14T18:50:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.