GPUSparse: GPU-Accelerated Learned Sparse Retrieval with Parallel Inverted Indices

• URL: http://arxiv.org/abs/2606.26441v1
• Date: Wed, 24 Jun 2026 23:07:57 GMT
• Title: GPUSparse: GPU-Accelerated Learned Sparse Retrieval with Parallel Inverted Indices
• Abstract summary: We present GPUSparse, a system for GPU-accelerated exact learned sparse retrieval.<n>A GPU-parallel inverted index with block-aligned, warp-coalesced posting lists is used.<n>Batched scatter-add scoring algorithm processes hundreds of queries simultaneously.
• Score: 3.3723515662362265
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learned sparse retrieval models such as SPLADE achieve retrieval quality competitive with dense models while preserving the interpretability and exact-match advantages of sparse representations. However, inference-time scoring still relies on CPU-bound inverted index traversal algorithms (WAND, Block-Max WAND), creating a fundamental bottleneck for real-time serving at scale. We present GPUSparse, a system for GPU-accelerated exact learned sparse retrieval that introduces: (1) a GPU-parallel inverted index with block-aligned, warp-coalesced posting lists; (2) a batched scatter-add scoring algorithm that processes hundreds of queries simultaneously; and (3) fused Triton kernels with an analysis of the tradeoff between work-efficiency and hardware utilization. On MS MARCO passage ranking (8.8M passages) with real SPLADE embeddings, GPUSparse matches CPU exact scoring to three decimals (MRR@10=0.383, equal to Pyserini SPLADE at this precision; Recall@1000>=0.999 vs. dense matmul, the residual from floating-point tie-breaking) while providing a 235x speedup over Pyserini CPU at 8.8M documents (1.27ms vs. 298ms per query). Compared to Seismic (the fastest CPU sparse retrieval system), which trades 25% recall for speed (R@1000=0.738 vs. 0.983 exact), GPUSparse achieves exact scoring at 787 QPS throughput (batch 500) on the full 8.8M collection, with 1.3ms per query. Our document-parallel kernel reaches 62.6% of H100 peak HBM bandwidth, revealing a fundamental work-efficiency vs. bandwidth-efficiency tradeoff in GPU sparse retrieval. The reformulation of sparse scoring as scatter-add over an inverted index is shared with SPARe's iterative mode; our contribution is its fused-kernel realization, which we measure to be 23-270x faster than a faithful SPARe iterative reimplementation.

Related papers

• SparseDitto: Customizing GPU Kernels for Different Sparsity Patterns with LLM-Based Agentic System [45.52961410486319]
  We present SparseDitto, an LLM-based system that constructs a GPU kernel for each matrix, operator, and target GPU.<n>SparseDitto supports SpMV, SpMM, and SpGEMM within a unified design framework.<n>Its generated SpMM kernels also accelerate full-batch GCN training by up to 3.39x.
  arXiv  Detail & Related papers  (2026-08-05T16:41:03Z)
• TileMaxSim: IO-Aware GPU MaxSim Scoring with Dimension Tiling and Fused Product Quantization [3.3723515662362265]
  Multi-vector retrieval models such as ColBERT achieve state-of-the-art accuracy through fine-grained token-level MaxSim scoring.<n>Existing GPU implementations leave most hardware performance unused.<n>We present TileMaxSim, a family of IO-aware Triton kernels that close this gap.
  arXiv  Detail & Related papers  (2026-06-24T23:03:56Z)
• Accelerating GPU Inference of Large Language Models with Moderately Unstructured Sparse Weight Matrices [18.428868416628017]
  This paper proposes an efficient inference method for large language models (LLMs) with moderate sparsity.<n>It achieves up to 1.64x-level speedup over SpInfer and up to 1.41x end-to-end speedups over FlashLLM.
  arXiv  Detail & Related papers  (2026-06-13T13:38:27Z)
• Fast Log-Domain Sinkhorn Optimal Transport with Warp-Level GPU Reductions [1.6679662639178268]
  We present FastSinkhorn, a lightweight, native implementation of the log-domain Sinkhorn algorithm.<n>Our solver operates entirely in the log-domain, enabling computation for regularization parameters as small as epsilon = 10-4 where standard-domain methods fail.<n>We validate our solver on image color transfer, 3D point cloud matching, and convergence analysis, demonstrating that native kernels with careful numerical treatment provide a practical and efficient foundation for large-scale optimal transport.
  arXiv  Detail & Related papers  (2026-04-04T16:06:27Z)
• FastLoop: Parallel Loop Closing with GPU-Acceleration in Visual SLAM [8.555551861128174]
  This paper presents FastLoop, a GPU-accelerated loop closing module to alleviate this computational complexity.<n>We identify key performance bottlenecks in the loop closing pipeline of visual SLAM and address them through parallel optimizations on the GPU.
  arXiv  Detail & Related papers  (2026-03-17T23:04:48Z)
• dParallel: Learnable Parallel Decoding for dLLMs [77.24184219948337]
  Diffusion large language models (dLLMs) offer parallel token prediction and lower inference latency.<n>Existing open-source models still require nearly token-length decoding steps to ensure performance.<n>We introduce dParallel, a simple and effective method that unlocks the inherent parallelism of dLLMs for fast sampling.
  arXiv  Detail & Related papers  (2025-09-30T16:32:52Z)
• GPU-Accelerated Interpretable Generalization for Rapid Cyberattack Detection and Forensics [0.0]
  IG mechanism recently published in IEEE Transactions on Information Forensics and Security delivers state-of-the-art, evidence-based intrusion detection.<n>We present IG-GPU, a PyTorch re-architecture that offloads all pairwise intersections and subset evaluations to commodity GPU.<n>In 15k-record NSL-KDD dataset, IG-GPU shows a 116-fold speed-up over the multi-core CPU implementation of IG.
  arXiv  Detail & Related papers  (2025-07-16T12:38:19Z)
• 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float [52.079202872069835]
  Large-scale AI models, such as Large Language Models (LLMs) and Diffusion Models (DMs) have grown rapidly in size.<n>We introduce Dynamic-Length Float (DFloat11), a compression framework that reduces LLM and DM size by 30% while preserving outputs that are bit-for-bit identical to the original model.
  arXiv  Detail & Related papers  (2025-04-15T22:38:38Z)
• Speedy MASt3R [68.47052557089631]
  MASt3R redefines image matching as a 3D task by leveraging DUSt3R and introducing a fast reciprocal matching scheme.<n>Fast MASt3R achieves a 54% reduction in inference time (198 ms to 91 ms per image pair) without sacrificing accuracy.<n>This advancement enables real-time 3D understanding, benefiting applications like mixed reality navigation and large-scale 3D scene reconstruction.
  arXiv  Detail & Related papers  (2025-03-13T03:56:22Z)
• Breaking the Memory Barrier: Near Infinite Batch Size Scaling for Contrastive Loss [59.835032408496545]
  We propose a tile-based strategy that partitions the contrastive loss calculation into arbitrary small blocks. We also introduce a multi-level tiling strategy to leverage the hierarchical structure of distributed systems. Compared to SOTA memory-efficient solutions, it achieves a two-order-of-magnitude reduction in memory while maintaining comparable speed.
  arXiv  Detail & Related papers  (2024-10-22T17:59:30Z)
• TorchSparse++: Efficient Training and Inference Framework for Sparse Convolution on GPUs [20.4238781638402]
  Sparse convolution plays a pivotal role in emerging workloads, including point cloud processing in AR/VR, autonomous driving, and graph understanding in recommendation systems. Existing GPU libraries offer two dataflow types for sparse convolution. We introduce TorchSparse++, a new GPU library that achieves the best of both worlds.
  arXiv  Detail & Related papers  (2023-10-25T21:02:38Z)
• Providing Meaningful Data Summarizations Using Examplar-based Clustering in Industry 4.0 [67.80123919697971]
  We show, that our GPU implementation provides speedups of up to 72x using single-precision and up to 452x using half-precision compared to conventional CPU algorithms. We apply our algorithm to real-world data from injection molding manufacturing processes and discuss how found summaries help with steering this specific process to cut costs and reduce the manufacturing of bad parts.
  arXiv  Detail & Related papers  (2021-05-25T15:55:14Z)
• IRLI: Iterative Re-partitioning for Learning to Index [104.72641345738425]
  Methods have to trade between obtaining high accuracy while maintaining load balance and scalability in distributed settings. We propose a novel approach called IRLI, which iteratively partitions the items by learning the relevant buckets directly from the query-item relevance data. We mathematically show that IRLI retrieves the correct item with high probability under very natural assumptions and provides superior load balancing.
  arXiv  Detail & Related papers  (2021-03-17T23:13:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.