StreamIndex: Memory-Bounded Compressed Sparse Attention via Streaming Top-k

• URL: http://arxiv.org/abs/2605.02568v1
• Date: Mon, 04 May 2026 13:19:29 GMT
• Title: StreamIndex: Memory-Bounded Compressed Sparse Attention via Streaming Top-k
• Abstract summary: DeepSeek-V3.2 and V4 introduce Compressed Sparse Attention (CSA): a lightning indexer (a learned scoring projection over compressed keys) scores them, the top-k are selected per query, and a sparse attention kernel reads only those.<n>We present StreamIndex, a Triton implementation of the CSA pipeline whose central component is a chunked partition-merge top-k driver that never materializes the full intermediate.<n>Our contribution targets the indexer step; we make no claim of a faster attention kernel or of real-checkpoint end-to-end behavior.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: DeepSeek-V3.2 and V4 introduce Compressed Sparse Attention (CSA): a lightning indexer (a learned scoring projection over compressed keys) scores them, the top-k are selected per query, and a sparse attention kernel reads only those. Public CSA implementations materialize a [B, S, H_I, T] FP32 score tensor before the top-k reduction. With H_I=64 indexer heads and the V4-Flash compression ratio m=4, that intermediate is 256 GB at sequence length S=65,536, exceeding any single-GPU high-bandwidth-memory (HBM) budget. We present StreamIndex, a Triton implementation of the CSA pipeline whose central component is a chunked partition-merge top-k driver that never materializes the full intermediate. On synthetic-but-realistic V4-shaped inputs at the indexer-step (layer) level on a single NVIDIA H200, the materialize path runs out of memory (OOMs) at S=65,536 with V4-Flash dimensions; StreamIndex runs the same indexer to S=1,048,576 with 6.21 GB peak HBM, a 32x regime extension. Set-overlap recall against the materialize ground truth is bit-exact at small S where both fit; across three 5-point design-space sweeps (chunk size, key-tile size, top-k), mean recall rounds to 1.0000 with min recall at least 0.9980 in every cell. The chunked driver composes with TileLang's pipelined attention kernel: at S=262,144 with V4-Flash dimensions, the materialize indexer paired with TileLang attention OOMs while the chunked indexer paired with the same attention runs in 1.97 s at 18.56 GB peak. Our contribution targets the indexer step; we make no claim of a faster attention kernel or of real-checkpoint end-to-end behavior. Code: https://github.com/RightNow-AI/StreamIndex.

Related papers

• VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention [47.30761996669105]
  We propose VC-Attention, a training-free low-bit attention framework.<n>It addresses both by smoothing value tokens by a probability Cast. V-Smooth reorders value tokens by lightweight clustering online, so the tokens in a fused hardware block quantize well together.<n>It maps log-domain scores directly to E4M3 probability codes with one fused multiply-add, eliminating the FP32 exponential and the format conversion.
  arXiv  Detail & Related papers  (2026-09-14T16:17:35Z)
• Unfolding the Leech Lattice: Fused Multi-Shell Decoding and VRAM Layouts for 2-Bit LLM Weights [8.550323073861415]
  Leech-lattice vector quantization holds the strongest reported 2-bit quality under its own evaluation protocol.<n>This paper supplies one and measures its serving cost for decode-phase GEMV at batch 1.
  arXiv  Detail & Related papers  (2026-09-02T14:26:06Z)
• FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention [77.12062766962815]
  Lookahead Sparse Attention (LSA) is powered by a Neural Memory Indexer built upon the DeepSeek-V4 architecture.<n>We instantiate this architecture via a backbone-free decoupled training strategy.<n>We show that FM-DS-V4 compresses the average physical KV cache footprint down to merely 13.5% of the full-context baseline.
  arXiv  Detail & Related papers  (2026-06-08T06:25:54Z)
• Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode [0.0]
  Physical AI systems run a different workload from cloud cloud LLM serving.<n>We measure batch-1 decode for three 7 to 8B-class GQA transformers across four NVIDIA GPUs.<n>The achieved fraction of peak bandwidth falls as peak bandwidth rises.
  arXiv  Detail & Related papers  (2026-05-28T21:03:14Z)
• MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference [75.41426145782751]
  We propose a drop-in replacement for the DSA indexer that treats its indexer heads as a pool of mixture-of-experts.<n> MISA matches the dense DSA indexer on LongBench across DeepSeek-V3.2 and GLM-5 while running with eight and four times fewer indexer heads respectively.<n>Our TileLang kernel delivers roughly a 3.82 times speedup over DSA's original indexer kernel on a single NVIDIA H200 GPU.
  arXiv  Detail & Related papers  (2026-05-08T07:19:34Z)
• Open-TQ-Metal: Fused Compressed-Domain Attention for Long-Context LLM Inference on Apple Silicon [0.0]
  We present Open-TQ-Metal, the first implementation of fused compressed-domain attention on Apple Silicon.<n>It enables 128K-context inference for Llama 3.1 70B on a single 64GB consumer Mac.<n>Open-TQ-Metal quantizes the KV cache to int4 on the fly and computes attention directly on the compressed representation.
  arXiv  Detail & Related papers  (2026-04-18T10:39:28Z)
• Breaking the KV Cache Bottleneck: Fan Duality Model Achieves O(1) Decode Memory with Superior Associative Recall [0.0]
  Fan Duality Model resolves tension between memory efficiency and associative recall in sequence modeling.<n>FDM separates sequence processing into two components: a wave component that compresses long-range patterns into a hidden state, and a particle component that retrieves specific tokens.<n>We propose Freeze-Scan, a two-phase training strategy that freezes the recurrent scan and optimize the cache jointly with embeddings.
  arXiv  Detail & Related papers  (2026-04-09T02:00:30Z)
• FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling [20.849774181776414]
  FlashAttention-4 achieves up to 1.3$times$ speedup over cuDNN 9.13 and 2.7$times$ over Triton on B200 GPUs with BF16.<n>We implement FlashAttention-4 entirely in CuTe- embedded in Python, achieving 20-30$times$ faster compile times compared to traditional C++ template-based approaches.
  arXiv  Detail & Related papers  (2026-03-05T18:24:49Z)
• FuseSampleAgg: Fused Neighbor Sampling and Aggregation for Mini-batch GNNs [51.56484100374058]
  FuseSampleAgg fuses neighbor and sampling mean aggregation into a single pass for GraphSAGE.<n>Operator is deterministic, integrates with standard PyTorchs, and ships with scripts that reproduce all tables and figures from CSV logs.
  arXiv  Detail & Related papers  (2025-11-17T17:57:18Z)
• FG-Attn: Leveraging Fine-Grained Sparsity In Diffusion Transformers [6.260564859775371]
  We propose FG-Attn, a sparse attention mechanism for long-context diffusion transformers.<n>Our approach skips computations at the granularity of Mx1 slices of the attention map.<n>It achieves an average 1.55X speedup for 5 second, 480p videos, and an average 1.41X for 5 second, 720p videos on a single H100 GPU.
  arXiv  Detail & Related papers  (2025-09-20T03:48:32Z)
• Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Index [110.90283601829724]
  We present infini-gram mini, a system that can make petabyte-level text corpora searchable.<n>Based on the FM-index data structure, our system creates indexes with size only 44% of the corpus.<n>We show one important use case in a large-scale analysis of benchmark contamination.
  arXiv  Detail & Related papers  (2025-06-13T21:13:57Z)
• vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving [53.972175896814505]
  Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests. Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests.
  arXiv  Detail & Related papers  (2024-07-22T14:37:58Z)
• DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Training [82.06732962485754]
  FlashAttention effectively reduces the quadratic peak memory usage to linear in training transformer-based large language models (LLMs) on a single GPU. We introduce DISTFLASHATTN, a memory-efficient attention mechanism optimized for long-context LLMs training. It achieves 1.67x and 1.26 - 1.88x speedup compared to recent Ring Attention and DeepSpeed-Ulysses.
  arXiv  Detail & Related papers  (2023-10-05T03:47:57Z)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness [80.3586155104237]
  FlashAttention is an IO-aware exact attention algorithm for Transformers. It reduces the number of memory reads/writes between GPU high bandwidth memory (HBM) and GPU on-chip. FlashAttention and block-sparse FlashAttention enable longer context in Transformers.
  arXiv  Detail & Related papers  (2022-05-27T17:53:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.