MixFormer: Linear Transformer with Mixture of Memory Experts

• URL: http://arxiv.org/abs/2608.09468v1
• Date: Mon, 10 Aug 2026 11:38:30 GMT
• Title: MixFormer: Linear Transformer with Mixture of Memory Experts
• Abstract summary: We propose MixFormer, a novel linear Transformer that integrates a Mixture-of-Memory-Experts (MoE) mechanism.<n>The model maintains differentiated memory states through multiple collaborating memory experts and employs a novel Time-Aware Linear Attention (TALA) mechanism.<n> Experiments on long-sequence text and image generation tasks demonstrate that MixFormer achieves significant performance gains.
• Score: 8.465156775656679
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: State Space Models (SSMs), as a mainstream research direction of linear Transformers, aim to achieve higher efficiency than standard Transformers in long-context modeling. However, existing SSMs suffer from limited input adaptivity and constrained memory capacity, leading to information loss when modeling ultra-long sequences. To address these limitations, we propose MixFormer, a novel linear Transformer that integrates a Mixture-of-Memory-Experts (MoE) mechanism. Specifically, the model maintains differentiated memory states through multiple collaborating memory experts and employs a novel Time-Aware Linear Attention (TALA) mechanism, which leverages learnable exponential decay functions and positional biases to dynamically update memory. This design enables the model to selectively reinforce important historical information while effectively mitigating memory dilution, substantially improving long-range dependency modeling. Experiments on long-sequence text and image generation tasks demonstrate that MixFormer not only achieves significant performance gains but also provides a more sustainable computational backbone for the next generation of web infrastructure.

Related papers

• Flash PD-SSM: Memory-Optimized Structured Sparse State-Space Models [62.362971723881095]
  State-space models (SSMs) face a fundamental trade-off between efficiency and expressivity.<n>We propose Flash PD-SSM, a novel SSM that achieves comparable throughput to widely-used structured SSMs with significantly better expressivity guarantees.<n>We show that Flash PD-SSM defines a new state-of-the-art (SoTA) accuracy among competing SSM methods.
  arXiv  Detail & Related papers  (2026-05-18T22:06:33Z)
• Linear-Time and Constant-Memory Text Embeddings Based on Recurrent Language Models [3.740517574718329]
  We introduce a vertically chunked inference strategy that enables fast embedding generation with memory usage that becomes constant in the input length once it exceeds the vertical chunk size.<n>By fine-tuning Mamba2 models, we demonstrate their viability as general-purpose text embedders.<n>We empirically validate the applicability of our inference strategy to Mamba2, RWKV, and xLSTM models, confirming consistent runtime-memory trade-offs across architectures.
  arXiv  Detail & Related papers  (2026-04-20T12:50:15Z)
• Memory Caching: RNNs with Growing Memory [56.25483647131372]
  We introduce Memory Caching (MC), a technique that enhances recurrent models by caching checkpoints of memory states (a.k.a. hidden states)<n>We propose four variants of MC, including gated aggregation and sparse selective mechanisms, and discuss their implications on both linear and deep memory modules.<n>The results indicate that while Transformers achieve the best accuracy, our MC variants show competitive performance, close the gap with Transformers, and performs better than state-of-the-art recurrent models.
  arXiv  Detail & Related papers  (2026-02-27T18:53:41Z)
• MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling [80.48332380100915]
  MiniCPM-SALA is a hybrid model that integrates the high-fidelity long-context modeling of sparse attention with the global efficiency of linear attention.<n>On a single NVIDIA A6000D GPU, the model achieves up to 3.5x the inference speed of the full-attention model at the sequence length of 256K tokens.
  arXiv  Detail & Related papers  (2026-02-12T09:37:05Z)
• MoVE: Mixture of Value Embeddings -- A New Axis for Scaling Parametric Memory in Autoregressive Models [0.9222161299777548]
  We introduce $textbfMoVE (Mixture of Value Embeddings)$, a mechanism that breaks the rigid structural coupling of model capacity to computational cost.<n>MoVE decouples memory from compute by introducing a global bank of learnable value embeddings shared across all attention layers.<n>We validate MoVE through strictly controlled experiments on two representative applications of autoregressive modeling: Text Generation and Image Generation.
  arXiv  Detail & Related papers  (2026-01-30T12:07:23Z)
• MS-SSM: A Multi-Scale State Space Model for Efficient Sequence Modeling [60.648359990090846]
  State-space models (SSMs) have recently attention as an efficient alternative to computationally expensive attention-based models for sequence modeling.<n>This paper introduces a multi-scale SSM framework that represents sequence dynamics across multiple resolution and processing each resolution with specialized state-space dynamics.
  arXiv  Detail & Related papers  (2025-12-29T19:36:28Z)
• MemMamba: Rethinking Memory Patterns in State Space Model [6.537535831000493]
  We show that selective state-space models such as Mamba have high efficiency with O(n) time and O(1) recurrent inference, yet their long-range memory decays exponentially.<n>Inspired by how humans distill and retain salient information when reading long documents, we propose MemMamba.<n>MemMamba achieves significant improvements over existing Mamba variants and Transformers on long-sequence benchmarks.
  arXiv  Detail & Related papers  (2025-09-28T14:40:58Z)
• Dynamic Memory-enhanced Transformer for Hyperspectral Image Classification [3.5093938502961763]
  Hyperspectral image (HSI) classification remains a challenging task due to the intricate spatial-spectral correlations.<n>Existing transformer models excel in capturing long-range dependencies but often suffer from information redundancy and attention inefficiencies.<n>MemFormer introduces a memory-enhanced multi-head attention mechanism that iteratively refines a dynamic memory module.<n>A dynamic memory enrichment strategy progressively captures complex spatial and spectral dependencies, leading to more expressive feature representations.
  arXiv  Detail & Related papers  (2025-04-17T17:43:34Z)
• Recurrent Action Transformer with Memory [39.58317527488534]
  This paper proposes a novel model architecture that incorporates a recurrent memory mechanism designed to regulate information retention. We conduct experiments on memory-intensive environments (ViZDoom-Two-Colors, T-Maze, Memory Maze, Minigrid-Memory), classic Atari games, and MuJoCo control environments. The results show that using memory can significantly improve performance in memory-intensive environments, while maintaining or improving results in classic environments.
  arXiv  Detail & Related papers  (2023-06-15T19:29:08Z)
• Blockwise Parallel Transformer for Large Context Models [70.97386897478238]
  Blockwise Parallel Transformer (BPT) is a blockwise computation of self-attention and feedforward network fusion to minimize memory costs. By processing longer input sequences while maintaining memory efficiency, BPT enables training sequences 32 times longer than vanilla Transformers and up to 4 times longer than previous memory-efficient methods.
  arXiv  Detail & Related papers  (2023-05-30T19:25:51Z)
• Memformer: A Memory-Augmented Transformer for Sequence Modeling [55.780849185884996]
  We present Memformer, an efficient neural network for sequence modeling. Our model achieves linear time complexity and constant memory space complexity when processing long sequences.
  arXiv  Detail & Related papers  (2020-10-14T09:03:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.