STanHop: Sparse Tandem Hopfield Model for Memory-Enhanced Time Series Prediction

• URL: http://arxiv.org/abs/2312.17346v1
• Date: Thu, 28 Dec 2023 20:26:23 GMT
• Title: STanHop: Sparse Tandem Hopfield Model for Memory-Enhanced Time Series Prediction
• Authors: Dennis Wu, Jerry Yao-Chieh Hu, Weijian Li, Bo-Yu Chen, Han Liu
• Abstract summary: We present a novel Hopfield-based neural network block, which sparsely learns and stores both temporal and cross-series representations. In essence, STanHop sequentially learn temporal representation and cross-series representation using two tandem sparse Hopfield layers. We show that our framework endows a tighter memory retrieval error compared to the dense counterpart without sacrificing memory capacity.
• Score: 13.815793371488613
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present STanHop-Net (Sparse Tandem Hopfield Network) for multivariate time series prediction with memory-enhanced capabilities. At the heart of our approach is STanHop, a novel Hopfield-based neural network block, which sparsely learns and stores both temporal and cross-series representations in a data-dependent fashion. In essence, STanHop sequentially learn temporal representation and cross-series representation using two tandem sparse Hopfield layers. In addition, StanHop incorporates two additional external memory modules: a Plug-and-Play module and a Tune-and-Play module for train-less and task-aware memory-enhancements, respectively. They allow StanHop-Net to swiftly respond to certain sudden events. Methodologically, we construct the StanHop-Net by stacking STanHop blocks in a hierarchical fashion, enabling multi-resolution feature extraction with resolution-specific sparsity. Theoretically, we introduce a sparse extension of the modern Hopfield model (Generalized Sparse Modern Hopfield Model) and show that it endows a tighter memory retrieval error compared to the dense counterpart without sacrificing memory capacity. Empirically, we validate the efficacy of our framework on both synthetic and real-world settings.

Related papers

• Hopfield-Fenchel-Young Networks: A Unified Framework for Associative Memory Retrieval [25.841394444834933]
  Associative memory models, such as Hopfield networks, have garnered renewed interest due to advancements in memory capacity and connections with self-attention in transformers. In this work, we introduce a unified framework-Hopfield-Fenchel-Young networks-which generalizes these models to a broader family of energy functions.
  arXiv  Detail & Related papers  (2024-11-13T13:13:07Z)
• Deciphering Movement: Unified Trajectory Generation Model for Multi-Agent [53.637837706712794]
  We propose a Unified Trajectory Generation model, UniTraj, that processes arbitrary trajectories as masked inputs. Specifically, we introduce a Ghost Spatial Masking (GSM) module embedded within a Transformer encoder for spatial feature extraction. We benchmark three practical sports game datasets, Basketball-U, Football-U, and Soccer-U, for evaluation.
  arXiv  Detail & Related papers  (2024-05-27T22:15:23Z)
• ColorMNet: A Memory-based Deep Spatial-Temporal Feature Propagation Network for Video Colorization [62.751303924391564]
  How to effectively explore spatial-temporal features is important for video colorization. We develop a memory-based feature propagation module that can establish reliable connections with features from far-apart frames. We develop a local attention module to aggregate the features from adjacent frames in a spatial-temporal neighborhood.
  arXiv  Detail & Related papers  (2024-04-09T12:23:30Z)
• Nonparametric Modern Hopfield Models [12.160725212848137]
  We present a nonparametric construction for deep learning compatible modern Hopfield models. Key contribution stems from interpreting the memory storage and retrieval processes in modern Hopfield models. We introduce textitsparse-structured modern Hopfield models with sub-quadratic complexity.
  arXiv  Detail & Related papers  (2024-04-05T05:46:20Z)
• BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Model [6.888608574535993]
  BiSHop handles the two major challenges of deep tabular learning: non-rotationally invariant data structure and feature sparsity in data. BiSHop uses a dual-component approach, sequentially processing data both column-wise and row-wise. We show BiSHop surpasses current SOTA methods with significantly less HPO runs.
  arXiv  Detail & Related papers  (2024-04-04T23:13:32Z)
• Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models [5.929540708452128]
  We propose a two-stage memory retrieval dynamics for modern Hopfield models. Key contribution is a learnable feature map $Phi$ which transforms the Hopfield energy function into kernel space. It utilizes the stored memory patterns as learning data to enhance memory capacity across all modern Hopfield models.
  arXiv  Detail & Related papers  (2024-04-04T23:05:30Z)
• FuTH-Net: Fusing Temporal Relations and Holistic Features for Aerial Video Classification [49.06447472006251]
  We propose a novel deep neural network, termed FuTH-Net, to model not only holistic features, but also temporal relations for aerial video classification. Our model is evaluated on two aerial video classification datasets, ERA and Drone-Action, and achieves the state-of-the-art results.
  arXiv  Detail & Related papers  (2022-09-22T21:15:58Z)
• Gait Recognition in the Wild with Multi-hop Temporal Switch [81.35245014397759]
  gait recognition in the wild is a more practical problem that has attracted the attention of the community of multimedia and computer vision. This paper presents a novel multi-hop temporal switch method to achieve effective temporal modeling of gait patterns in real-world scenes.
  arXiv  Detail & Related papers  (2022-09-01T10:46:09Z)
• Differentiable Point-Based Radiance Fields for Efficient View Synthesis [57.56579501055479]
  We propose a differentiable rendering algorithm for efficient novel view synthesis. Our method is up to 300x faster than NeRF in both training and inference. For dynamic scenes, our method trains two orders of magnitude faster than STNeRF and renders at near interactive rate.
  arXiv  Detail & Related papers  (2022-05-28T04:36:13Z)
• Memory-Guided Semantic Learning Network for Temporal Sentence Grounding [55.31041933103645]
  We propose a memory-augmented network that learns and memorizes the rarely appeared content in TSG tasks. MGSL-Net consists of three main parts: a cross-modal inter-action module, a memory augmentation module, and a heterogeneous attention module.
  arXiv  Detail & Related papers  (2022-01-03T02:32:06Z)
• Hopfield Networks is All You Need [8.508381229662907]
  We introduce a modern Hopfield network with continuous states and a corresponding update rule. The new Hopfield network can store exponentially (with the dimension of the associative space) many patterns, retrieves the pattern with one update, and has exponentially small retrieval errors. We demonstrate the broad applicability of the Hopfield layers across various domains.
  arXiv  Detail & Related papers  (2020-07-16T17:52:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.