Asymmetric Virtual Memory Paging for Hybrid Mamba-Transformer Inference

• URL: http://arxiv.org/abs/2605.22416v1
• Date: Thu, 21 May 2026 12:37:34 GMT
• Title: Asymmetric Virtual Memory Paging for Hybrid Mamba-Transformer Inference
• Abstract summary: Hybrid language models like Jamba mix attention layers with State Space Models (SSMs)<n>Unified pools pad SSM states to attention page sizes, wasting up to 7.3x capacity.<n>Asymmetric Virtual Memory Paging (AVMP) separates the two cache types into physically distinct pools behind a unified virtual address space.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hybrid language models like Jamba mix attention layers with State Space Models (SSMs), creating two memory cache types with opposite profiles: Key-Value (KV) caches grow linearly with sequence length, while SSM states stay fixed per layer. Current inference engines handle this poorly. Unified pools pad SSM states to attention page sizes, wasting up to 7.3x capacity. Static dual pools cannot adapt when prompt distributions shift between requests. We present Asymmetric Virtual Memory Paging (AVMP). The allocator separates the two cache types into physically distinct pools behind a unified virtual address space, and migrates capacity between pools when one runs out. Migration triggers only on allocation failure, keeping behavior deterministic. We evaluate AVMP across 270 synthetic cells plus 60 cells of ShareGPT trace replay on an RTX 3060 12GB. Out-of-Memory events drop 7.6% and request throughput improves 1.83x to 13.3x across synthetic workloads and 2.36x on ShareGPT. All gains hold under paired-bootstrap 95% confidence intervals. A phase-time breakdown reveals two distinct mechanisms: shorter OOM recovery on capacity-pressured workloads, and faster allocation calls on KV-heavy workloads. Implementation is pure Python; Triton integration is future work.

Related papers

• Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale [18.318746660954663]
  Block diffusion restores caching by decoding block-by-block.<n>We study this recipe at scale, pretraining three 3B block-diffusion denoisers on 300B tokens.
  arXiv  Detail & Related papers  (2026-09-09T23:37:47Z)
• Dual-Grained Agent Memory and Shapley Context Attribution for Multimodal Agentic Learner [55.80248281761638]
  multimodal large language models (MLLMs) deliver impressive perception yet still falter on scientific and mathematical reasoning.<n>We propose DG-Mem, a dual-grained agentic memory framework that augments a frozen MLLM with a non- externally stored memory built once from training-time rollouts and consulted read-only at test time.
  arXiv  Detail & Related papers  (2026-08-24T13:56:01Z)
• ConvMemory v3: A Validity Context Layer for Conversational Memory via Target-Conditioned Relation Verification [0.0]
  ConvMemory v3 adds a validity context layer that detects and surfaces this update evidence.<n>The core mechanism is a dual-evidence gate that conditions a relation judgment on the specific target proposition.<n>On a synthetic multi-hop validity benchmark the gate reaches 90.12% +/- 1.73 accuracy.
  arXiv  Detail & Related papers  (2026-06-25T08:36:43Z)
• Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion [61.57938553036056]
  ARL2 is a hybrid attention module that replaces quadratic cross-frame attention with a fixed-size recurrent state.<n>We decompose self-attention into two branches: an intra-frame softmax branch for spatial detail and local dependencies, and an inter-frame recurrent linear branch that maintains a fixed-size state for streaming context.<n>With 75% of layers replaced by hybrid linear attention, the model achieves up to 2.26 wall-clock speedup and 54% memory reduction, while maintaining comparable quality with improving temporal consistency.
  arXiv  Detail & Related papers  (2026-05-15T19:33:45Z)
• Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference [0.0]
  Key-value ( KV) cache memory management is the primary bottleneck in large-scale GPU inference serving.<n>Current systems suffer from three compounding inefficiencies.<n>We present a unified system that addresses all three problems.
  arXiv  Detail & Related papers  (2026-04-19T21:34:09Z)
• Once-for-All Channel Mixers (HYPERTINYPW): Generative Compression for TinyML [0.0]
  We present HYPER-TINYPW, a compression-as-generation approach that replaces most stored PW weights with generated weights.<n>A shared micro-MLP synthesizes PW kernels once at load time from tiny per-layer codes, caches them, and executes them with standard integer operators.<n>It preserves commodity MCU runtimes and adds only a one-off cost.
  arXiv  Detail & Related papers  (2026-03-26T01:08:52Z)
• PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cache [61.57938553036056]
  We introduce PackCache, a training-free KV-cache management method that compacts the KV cache through three coordinated mechanisms.<n>In terms of efficiency, PackCache accelerates end-to-end generation by 1.7-2.2x on 48-frame long sequences.
  arXiv  Detail & Related papers  (2026-01-07T19:51:06Z)
• MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models [58.3342517278868]
  This paper describes the design of Mixed-precision AutoRegressive LINear kernels. It shows that batchsizes up to 16-32 can be supported with close to maximum ($4times$) quantization speedup. MarLIN accomplishes this via a combination of techniques, such as asynchronous memory access, complex task scheduling and pipelining.
  arXiv  Detail & Related papers  (2024-08-21T16:10:41Z)
• vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving [53.972175896814505]
  Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests. Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests.
  arXiv  Detail & Related papers  (2024-07-22T14:37:58Z)
• B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memory [91.81390121042192]
  We develop a class of models called B'MOJO to seamlessly combine eidetic and fading memory within an composable module. B'MOJO's ability to modulate eidetic and fading memory results in better inference on longer sequences tested up to 32K tokens.
  arXiv  Detail & Related papers  (2024-07-08T18:41:01Z)
• Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference [1.9639467358416092]
  Transformers have emerged as the backbone of large language models (LLMs) We propose Dynamic Memory Compression (DMC), a method for online key-value cache compression at inference time. We retrofit pre-trained LLMs such as Llama 2 (7B, 13B and 70B) into DMC Transformers, achieving up to 7x throughput increase during auto-regressive inference on an NVIDIA H100 GPU.
  arXiv  Detail & Related papers  (2024-03-14T17:59:26Z)
• GEMEL: Model Merging for Memory-Efficient, Real-Time Video Analytics at the Edge [10.276140547573437]
  We present model merging, a new memory management technique that exploits architectural similarities between edge vision models. Experiments across diverse workloads reveal that GEMEL reduces memory usage by up to 60.7%, and improves overall accuracy by 8-39% relative to time/space sharing alone.
  arXiv  Detail & Related papers  (2022-01-19T16:45:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.