Invertible Memory Flow Networks

• URL: http://arxiv.org/abs/2602.00535v1
• Date: Sat, 31 Jan 2026 06:04:40 GMT
• Title: Invertible Memory Flow Networks
• Authors: Liyu Zerihun, Alexandr Plashchinsky,
• Abstract summary: Invertible Memory Flow Networks (IMFN) make long sequence compression tractable.<n>We decompose the problem into pairwise merges using a binary tree of "sweeper" modules.<n>For online inference, we distilled into a constant-cost recurrent student achieving O(1) sequential steps.
• Score: 45.88028371034407
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long sequence neural memory remains a challenging problem. RNNs and their variants suffer from vanishing gradients, and Transformers suffer from quadratic scaling. Furthermore, compressing long sequences into a finite fixed representation remains an intractable problem due to the difficult optimization landscape. Invertible Memory Flow Networks (IMFN) make long sequence compression tractable through factorization: instead of learning end-to-end compression, we decompose the problem into pairwise merges using a binary tree of "sweeper" modules. Rather than learning to compress long sequences, each sweeper learns a much simpler 2-to-1 compression task, achieving O(log N) depth with sublinear error accumulation in sequence length. For online inference, we distilled into a constant-cost recurrent student achieving O(1) sequential steps. Empirical results validate IMFN on long MNIST sequences and UCF-101 videos, demonstrating compression of high-dimensional data over long sequences.

Related papers

• UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression [86.33995240043936]
  UniGist is a sequence-level long-context compression framework for large language models.<n>It efficiently preserves context information by replacing raw tokens with special compression tokens (gists) in a fine-grained manner.<n>Our scheme also supports flexible inference by allowing the actual removal of compressed tokens, resulting in real-time memory savings.
  arXiv  Detail & Related papers  (2025-09-19T08:47:37Z)
• mGRADE: Minimal Recurrent Gating Meets Delay Convolutions for Lightweight Sequence Modeling [0.5236468296934584]
  mGRADE is a hybrid-memory system that integrates a temporal 1D-convolution with learnable spacings followed by a minimal gated recurrent unit.<n>We demonstrate that mGRADE effectively separates and preserves multi-scale temporal features.<n>This highlights mGRADE's promise as an efficient solution for memory-constrained multi-scale temporal processing at the edge.
  arXiv  Detail & Related papers  (2025-07-02T15:44:35Z)
• Lattice: Learning to Efficiently Compress the Memory [13.765057453744427]
  This paper introduces Lattice, a novel recurrent neural network (RNN) mechanism that efficiently compress the cache into a fixed number of memory slots.<n>We formulate this compression as an online optimization problem and derive a dynamic memory update rule based on a single gradient descent step.<n>The experimental results show that Lattice achieves the best perplexity compared to all baselines across diverse context lengths.
  arXiv  Detail & Related papers  (2025-04-08T03:48:43Z)
• Mini-Sequence Transformer: Optimizing Intermediate Memory for Long Sequences Training [78.93900796545523]
  Mini-Sequence Transformer (MsT) is a methodology for highly efficient and accurate LLM training with extremely long sequences. MsT partitions input sequences and iteratively processes mini-sequences to reduce intermediate memory usage. integrated with the huggingface library, MsT successfully extends the maximum context length of Qwen, Mistral, and Gemma-2 by 12-24x.
  arXiv  Detail & Related papers  (2024-07-22T01:52:30Z)
• Blockwise Parallel Transformer for Large Context Models [70.97386897478238]
  Blockwise Parallel Transformer (BPT) is a blockwise computation of self-attention and feedforward network fusion to minimize memory costs. By processing longer input sequences while maintaining memory efficiency, BPT enables training sequences 32 times longer than vanilla Transformers and up to 4 times longer than previous memory-efficient methods.
  arXiv  Detail & Related papers  (2023-05-30T19:25:51Z)
• Sequence Modeling with Multiresolution Convolutional Memory [27.218134279968062]
  We build a new building block for sequence modeling called a MultiresLayer. The key component of our model is the multiresolution convolution, capturing multiscale trends in the input sequence. Our model yields state-of-the-art performance on a number of sequence classification and autoregressive density estimation tasks.
  arXiv  Detail & Related papers  (2023-05-02T17:50:54Z)
• Latent Discretization for Continuous-time Sequence Compression [21.062288207034968]
  In this work, we treat data sequences as observations from an underlying continuous-time process. We show that our approaches can automatically achieve reductions in bit rates by learning how to discretize.
  arXiv  Detail & Related papers  (2022-12-28T01:15:27Z)
• DBA: Efficient Transformer with Dynamic Bilinear Low-Rank Attention [53.02648818164273]
  We present an efficient yet effective attention mechanism, namely the Dynamic Bilinear Low-Rank Attention (DBA) DBA compresses the sequence length by input-sensitive dynamic projection matrices and achieves linear time and space complexity. Experiments over tasks with diverse sequence length conditions show that DBA achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-11-24T03:06:36Z)
• Short-Term Memory Optimization in Recurrent Neural Networks by Autoencoder-based Initialization [79.42778415729475]
  We explore an alternative solution based on explicit memorization using linear autoencoders for sequences. We show how such pretraining can better support solving hard classification tasks with long sequences. We show that the proposed approach achieves a much lower reconstruction error for long sequences and a better gradient propagation during the finetuning phase.
  arXiv  Detail & Related papers  (2020-11-05T14:57:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.