mGRADE: Minimal Recurrent Gating Meets Delay Convolutions for Lightweight Sequence Modeling

• URL: http://arxiv.org/abs/2507.01829v1
• Date: Wed, 02 Jul 2025 15:44:35 GMT
• Title: mGRADE: Minimal Recurrent Gating Meets Delay Convolutions for Lightweight Sequence Modeling
• Authors: Tristan Torchet, Christian Metzner, Laura Kriener, Melika Payvand,
• Abstract summary: mGRADE is a hybrid-memory system that integrates a temporal 1D-convolution with learnable spacings followed by a minimal gated recurrent unit.<n>We demonstrate that mGRADE effectively separates and preserves multi-scale temporal features.<n>This highlights mGRADE's promise as an efficient solution for memory-constrained multi-scale temporal processing at the edge.
• Score: 0.5236468296934584
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Edge devices for temporal processing demand models that capture both short- and long- range dynamics under tight memory constraints. While Transformers excel at sequence modeling, their quadratic memory scaling with sequence length makes them impractical for such settings. Recurrent Neural Networks (RNNs) offer constant memory but train sequentially, and Temporal Convolutional Networks (TCNs), though efficient, scale memory with kernel size. To address this, we propose mGRADE (mininally Gated Recurrent Architecture with Delay Embedding), a hybrid-memory system that integrates a temporal 1D-convolution with learnable spacings followed by a minimal gated recurrent unit (minGRU). This design allows the convolutional layer to realize a flexible delay embedding that captures rapid temporal variations, while the recurrent module efficiently maintains global context with minimal memory overhead. We validate our approach on two synthetic tasks, demonstrating that mGRADE effectively separates and preserves multi-scale temporal features. Furthermore, on challenging pixel-by-pixel image classification benchmarks, mGRADE consistently outperforms both pure convolutional and pure recurrent counterparts using approximately 20% less memory footprint, highlighting its suitability for memory-constrained temporal processing at the edge. This highlights mGRADE's promise as an efficient solution for memory-constrained multi-scale temporal processing at the edge.

Related papers

• Neural Cellular Automata: From Cells to Pixels [66.28419836820035]
  Bio-inspired Neural Cellular Automata (NCAs) are bio-inspired systems in which identical cells self-organize to form complex and coherent patterns by applying simple local rules.<n>NCAs display striking emergent behaviors including self-regeneration, generalization and robustness to unseen situations, and spontaneous motion.<n>We show that NCAs equipped with our implicit decoder can generate full-HD outputs in real time while preserving their self-organizing, emergent properties.
  arXiv  Detail & Related papers  (2025-06-28T14:30:21Z)
• Long-Context State-Space Video World Models [66.28743632951218]
  We propose a novel architecture leveraging state-space models (SSMs) to extend temporal memory without compromising computational efficiency.<n>Central to our design is a block-wise SSM scanning scheme, which strategically trades off spatial consistency for extended temporal memory.<n>Experiments on Memory Maze and Minecraft datasets demonstrate that our approach surpasses baselines in preserving long-range memory.
  arXiv  Detail & Related papers  (2025-05-26T16:12:41Z)
• Lattice: Learning to Efficiently Compress the Memory [13.765057453744427]
  This paper introduces Lattice, a novel recurrent neural network (RNN) mechanism that efficiently compress the cache into a fixed number of memory slots.<n>We formulate this compression as an online optimization problem and derive a dynamic memory update rule based on a single gradient descent step.<n>The experimental results show that Lattice achieves the best perplexity compared to all baselines across diverse context lengths.
  arXiv  Detail & Related papers  (2025-04-08T03:48:43Z)
• MoM: Linear Sequence Modeling with Mixture-of-Memories [9.665802842933209]
  We introduce a novel architecture called Mixture-of-Memories (MoM)<n>MoM utilizes multiple independent memory states, with a router network directing input tokens to specific memory states.<n>MoM performs exceptionally well on recall-intensive tasks, surpassing existing linear sequence modeling techniques.
  arXiv  Detail & Related papers  (2025-02-19T12:53:55Z)
• Mini-Sequence Transformer: Optimizing Intermediate Memory for Long Sequences Training [78.93900796545523]
  Mini-Sequence Transformer (MsT) is a methodology for highly efficient and accurate LLM training with extremely long sequences. MsT partitions input sequences and iteratively processes mini-sequences to reduce intermediate memory usage. integrated with the huggingface library, MsT successfully extends the maximum context length of Qwen, Mistral, and Gemma-2 by 12-24x.
  arXiv  Detail & Related papers  (2024-07-22T01:52:30Z)
• MF-NeRF: Memory Efficient NeRF with Mixed-Feature Hash Table [62.164549651134465]
  We propose MF-NeRF, a memory-efficient NeRF framework that employs a Mixed-Feature hash table to improve memory efficiency and reduce training time while maintaining reconstruction quality. Our experiments with state-of-the-art Instant-NGP, TensoRF, and DVGO, indicate our MF-NeRF could achieve the fastest training time on the same GPU hardware with similar or even higher reconstruction quality.
  arXiv  Detail & Related papers  (2023-04-25T05:44:50Z)
• GLEAM: Greedy Learning for Large-Scale Accelerated MRI Reconstruction [50.248694764703714]
  Unrolled neural networks have recently achieved state-of-the-art accelerated MRI reconstruction. These networks unroll iterative optimization algorithms by alternating between physics-based consistency and neural-network based regularization. We propose Greedy LEarning for Accelerated MRI reconstruction, an efficient training strategy for high-dimensional imaging settings.
  arXiv  Detail & Related papers  (2022-07-18T06:01:29Z)
• Self-Gated Memory Recurrent Network for Efficient Scalable HDR Deghosting [59.04604001936661]
  We propose a novel recurrent network-based HDR deghosting method for fusing arbitrary length dynamic sequences. We introduce a new recurrent cell architecture, namely Self-Gated Memory (SGM) cell, that outperforms the standard LSTM cell. The proposed approach achieves state-of-the-art performance compared to existing HDR deghosting methods quantitatively across three publicly available datasets.
  arXiv  Detail & Related papers  (2021-12-24T12:36:33Z)
• Memformer: A Memory-Augmented Transformer for Sequence Modeling [55.780849185884996]
  We present Memformer, an efficient neural network for sequence modeling. Our model achieves linear time complexity and constant memory space complexity when processing long sequences.
  arXiv  Detail & Related papers  (2020-10-14T09:03:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.