SURGE: SuperBatch Unified Resource-efficient GPU Encoding for Heterogeneous Partitioned Data

• URL: http://arxiv.org/abs/2605.01060v1
• Date: Fri, 01 May 2026 19:51:50 GMT
• Title: SURGE: SuperBatch Unified Resource-efficient GPU Encoding for Heterogeneous Partitioned Data
• Abstract summary: SURGE is a streaming encoding system deployed in production to generate embeddings for over 800 million texts across 40,000 logical partitions.<n>On 10M texts with 4 NVIDIA L4768, SURGE delivers 26,413 texts/s -- matching fixed-times throughput using 12.6times less memory.
• Score: 3.1624024957575982
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present SURGE, a streaming GPU encoding system deployed in production to generate embeddings for over 800 million texts across 40,000 logical partitions. Production embedding pipelines face a tension between logical data partitioning and efficient GPU utilization: processing each partition independently incurs $P$ inter-process communication (IPC) calls whose overhead limits throughput for compute-light models. Our contributions are analytical: (i) a cost model (Theorem 1) predicting throughput within 2% across three encoders spanning a 15$\times$ parameter range; (ii) a memory-safety bound (Lemma 3) enabling a streaming two-threshold policy with peak memory $O(B_{\min} + n_{\max})$ rather than $O(N)$; and (iii) a $φ$/CV decision framework characterizing when the pattern applies beyond our workload. The naive fix of batching at fixed size requires $O(N)$ peak memory (32.7 GB at 10M texts; infeasible beyond ~60M on 192 GB nodes), produces no output until all encoding completes, and offers no fault tolerance. SURGE achieves the same throughput with $O(B_{\min} + n_{\max})$ bounded memory (2.6 GB), 68$\times$ faster time-to-first-output, and crash recovery at SuperBatch granularity. On 10M texts with 4 NVIDIA L4 GPUs, SURGE delivers 26,413 texts/s -- matching fixed-batch throughput while using 12.6$\times$ less memory. We validate on bge-base (109M, $d$=768, error 1.3%) and across log-normal $σ$ in {1.0, 1.72, 2.5} (speedup invariant within $\pm$3%), and compare against a partition-batched baseline (PB-PBP-LB), against which SURGE retains a 7% throughput edge and 2.5$\times$ faster TTFO. Complementary engineering -- zero-copy Arrow serialization (22-25$\times$ speedup) and async I/O pipelining (up to 93% benefit) -- realizes the design but is not the contribution.

Related papers

• Reducing Data Movement in the Galerkin Product of Block Algebraic Multigrid on GPUs [0.0]
  For 3D elasticity the fine operator has $3times3$ blocks, the sparse prolongator $3times6$, and the coarse operator $6times6$, a shape no vendor sparse library supports.<n>We present a sharedmemory-tiled kernel with a sorted search-free schedule that moves fewer bytes in less than half the time of the portable Kokkos team kernels.
  arXiv  Detail & Related papers  (2026-07-30T23:17:29Z)
• ChunkFT: Byte-Streamed Optimization for Memory-Efficient Full Fine-Tuning [58.54940026861599]
  textscChunkFT is a memory-efficient fine-tuning framework.<n>textscChunkFT enables gradient computation for arbitrary sub-tensors without modifying the network architecture.<n>textscChunkFT consistently outperforms existing memory-efficient baselines.
  arXiv  Detail & Related papers  (2026-05-20T13:44:44Z)
• A Controlled Study of Memory Hierarchy Transitions in Quantum Circuit Simulation on Apple M4 Pro Unified Memory Architecture [0.0]
  State-vector quantum circuit simulation is memory-bandwidth bound.<n>We address this using the Apple M4 Pro Unified Memory Architecture.<n>We show that peak streaming bandwidth does not predict simulation speedup for non-contiguous memory access patterns.
  arXiv  Detail & Related papers  (2026-05-09T08:22:55Z)
• Copy-as-Decode: Grammar-Constrained Parallel Prefill for LLM Editing [2.6382975801439836]
  LLMs edit text and code by autoregressively regenerating the full output, even when most tokens appear verbatim in the input.<n>We study Copy-as-Decode, a decoding-layer mechanism that recasts edit generation as structured decoding over a two-primitive grammar.
  arXiv  Detail & Related papers  (2026-04-20T12:29:53Z)
• Scaling DoRA: High-Rank Adaptation via Factored Norms and Fused Kernels [83.99688944263843]
  Weight-De Low-Rank Adaptation (DoRA) extends LoRA by decoupling weight magnitude from direction.<n>At d_in = 8192 and rank r = 384, a single module's norm requires about 512 MB of transient working memory in bf16.<n>A factored norm decomposes the squared norm into base, cross, and Gram terms computable through O(d_out r + r2) intermediates, eliminating the dense product.
  arXiv  Detail & Related papers  (2026-03-23T17:57:24Z)
• A Persistent-State Dataflow Accelerator for Memory-Bound Linear Attention Decode on FPGA [10.452946241750562]
  Gated DeltaNet (GDN) is a linear attention mechanism that replaces the growing KV cache with a fixed-size recurrent state.<n>We show that this bottleneck is architectural, not algorithmic, as all subquadratic sequence models exhibit arithmetic intensities below 1 FLOP/B at decode time.<n>We present an FPGA accelerator that eliminates this bottleneck by holding the full 2 MB recurrent state persistently in on-chip BRAM.
  arXiv  Detail & Related papers  (2026-03-06T06:03:38Z)
• FuseSampleAgg: Fused Neighbor Sampling and Aggregation for Mini-batch GNNs [51.56484100374058]
  FuseSampleAgg fuses neighbor and sampling mean aggregation into a single pass for GraphSAGE.<n>Operator is deterministic, integrates with standard PyTorchs, and ships with scripts that reproduce all tables and figures from CSV logs.
  arXiv  Detail & Related papers  (2025-11-17T17:57:18Z)
• 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float [52.079202872069835]
  Large-scale AI models, such as Large Language Models (LLMs) and Diffusion Models (DMs) have grown rapidly in size.<n>We introduce Dynamic-Length Float (DFloat11), a compression framework that reduces LLM and DM size by 30% while preserving outputs that are bit-for-bit identical to the original model.
  arXiv  Detail & Related papers  (2025-04-15T22:38:38Z)
• Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference [23.633481089469836]
  Auto-regressive decoding of Large Language Models (LLMs) results in significant overheads in their hardware performance.<n>We propose a novel parallel prompt decoding that requires only $0.0002$% trainable parameters, enabling efficient training on a single A100-40GB GPU in just 16 hours.<n>Our approach demonstrates up to 2.49$times$ speedup and maintains a minimal memory overhead of just $0.0004$%.
  arXiv  Detail & Related papers  (2024-05-28T22:19:30Z)
• Scalable 3D Registration via Truncated Entry-wise Absolute Residuals [65.04922801371363]
  A $3$D registration approach can process more than ten million ($107$) point pairs with over $99%$ random outliers. We call our method TEAR, as it involves minimizing an outlier-robust loss that computes Truncated Entry-wise Absolute Residuals.
  arXiv  Detail & Related papers  (2024-04-01T04:43:39Z)
• H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models [110.06476624089679]
  We introduce a novel approach for implementing the KV cache which significantly reduces its memory footprint. Our approach is based on the observation that a small portion of tokens contributes most of the value when computing attention scores. We propose Heavy Hitter (H$$O), a KV cache eviction policy that dynamically retains a balance of recent and H$$ tokens.
  arXiv  Detail & Related papers  (2023-06-24T20:11:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.