GRATING: Low-Latency and Memory-Efficient Semantic Selection on Device

• URL: http://arxiv.org/abs/2510.15620v1
• Date: Fri, 17 Oct 2025 13:06:09 GMT
• Title: GRATING: Low-Latency and Memory-Efficient Semantic Selection on Device
• Abstract summary: Top-K selection with cross-encoder rerankers underpins on-device AI services, such as retrieval-augmented generation, agent memory, and personalized recommendation.<n>Revisiting the objective of top-K selection, we reveal that only relative rankings matter, not exact per-candidate scores.<n>We propose monolithic forwarding and develop a training-free inference system, GRATING.
• Score: 1.7847219315073477
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Semantic top-K selection with cross-encoder rerankers underpins of on-device AI services, such as retrieval-augmented generation, agent memory, and personalized recommendation. However, its latency and memory demands dominate end-to-end budgets on edge hardware. Revisiting the objective of top-K selection, we reveal that only relative rankings matter, not exact per-candidate scores. We further observe sequence-level sparsity: relative rankings stabilize early in intermediate layers, allowing pruning opportunities prior to completing full inference. Building on this insight, we propose monolithic forwarding and develop a training-free inference system, GRATING. By maintaining a global view of all candidates, it reduces latency through progressive cluster pruning. It also bounds peak memory usage by strategically overlapping I/O with computation via dual-layer sliding window and chunked execution. We evaluate GRATING against state-of-the-art baselines on rerankers from 0.6B to 8B parameters across Apple M2 and RTX 5070. GRATING consistently reduces latency by up to 89.0% and peak memory by up to 94.9% in microbenchmarks, without any loss in precision. Across three real-world on-device AI applications, GRATING lowers latency by 11.6%-51.0% and peak memory by 18.6%-77.8%, demonstrating substantial improvements in efficiency and deployability.

Related papers

• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Do Newer Lightweight CNNs Perform Better Under Resource Constraints? A Controlled Multigenerational Study of Architecture, Initialization, Training Budget, and Efficiency [0.0]
  This study compares nine lightweight CNN model packages across CIFAR-10, CIFAR-100, and Tiny ImageNet.<n>We report top-1 accuracy, macro F1, top-5 accuracy, parameter count, FP32 storage, GMACs, batch-size-1 latency on an NVIDIA L4 and AMD 5 5500U CPU.
  arXiv  Detail & Related papers  (2026-07-02T10:14:07Z)
• A Survey of Spatial Memory Representations for Efficient Robot Navigation [4.560386676154887]
  Survey examines the spatial memory efficiency problem across 88 references spanning 52 systems.<n>We introduce the $= M_textpeak / M_textmap$, the ratio of peak runtime memory (the total RAM or GPU memory consumed during operation) to saved map size.
  arXiv  Detail & Related papers  (2026-04-13T02:12:17Z)
• MemMachine: A Ground-Truth-Preserving Memory System for Personalized AI Agents [2.541923091180284]
  Large Language Model (LLM) agents require persistent memory to maintain personalization, factual continuity, and long-horizon reasoning.<n>We present MemMachine, an open-source memory system that integrates short-term, long-term episodic, and profile memory.<n>MemMachine uses contextualized retrieval that expands nucleus matches with surrounding context, improving recall when relevant evidence spans multiple dialogue turns.
  arXiv  Detail & Related papers  (2026-04-06T16:57:06Z)
• Pip-Stereo: Progressive Iterations Pruner for Iterative Optimization based Stereo Matching [2.226667542457085]
  We analyze iterative refinement and reveal that disparity updates are spatially sparse and temporally redundant.<n>We propose a collaborative monocular prior transfer framework that implicitly embeds depth priors without requiring a dedicated monocular encoder.<n>Third, we develop FlashGRU, a hardware-aware RNN operator leveraging structured sparsity and I/O-conscious design.
  arXiv  Detail & Related papers  (2026-02-24T02:51:37Z)
• iResolveX: Multi-Layered Indirect Call Resolution via Static Reasoning and Learning-Augmented Refinement [9.184762707302099]
  Indirect call resolution is a key challenge in reverse engineering and control-flow graph recovery.<n>We present iResolveX, a framework that combines conservative static analysis with learning-based refinement.<n>iResolveX supports both conservative, recall-preserving and F1-optimized configurations and outperforms state-of-the-art systems.
  arXiv  Detail & Related papers  (2026-01-25T15:42:34Z)
• A Dynamic Retrieval-Augmented Generation System with Selective Memory and Remembrance [0.0]
  We introduce emph Adaptive RAG Memory (ARM), a retrieval-augmented generation (RAG) framework that replaces a static vector index with a emphdynamic memory substrate.<n>ARM reaches near state-of-the-art performance on a lightweight retrieval benchmark.<n>ARM yields competitive accuracy, self-regularizing memory growth, and interpretable retention dynamics without retraining the generatorcolorblack and provides practical trade-off between quality, latency and memory efficiency for production and research RAG system.
  arXiv  Detail & Related papers  (2026-01-04T21:51:41Z)
• Adaptive Execution Scheduler for DataDios SmartDiff [0.0]
  We present an adaptive scheduler for a single differencing engine (SmartDiff) with two execution modes.<n>The scheduler tunes batch size and worker/thread count within fixed CPU and memory budgets to minimize p95 latency.<n>Across synthetic and public benchmarks, the scheduler reduces p95 latency by 23 to 28 percent versus a tuned warm-up.
  arXiv  Detail & Related papers  (2025-10-09T05:40:16Z)
• vCache: Verified Semantic Prompt Caching [95.16654660556975]
  This paper proposes vCache, the first verified semantic cache with user-defined error rate guarantees.<n>It employs an online learning algorithm to estimate an optimal threshold for each cached prompt, enabling reliable cache responses without additional training.<n>Our experiments show that vCache consistently meets the specified error bounds while outperforming state-of-the-art static-threshold and fine-tuned embedding baselines.
  arXiv  Detail & Related papers  (2025-02-06T04:16:20Z)
• Breaking the Memory Barrier: Near Infinite Batch Size Scaling for Contrastive Loss [59.835032408496545]
  We propose a tile-based strategy that partitions the contrastive loss calculation into arbitrary small blocks. We also introduce a multi-level tiling strategy to leverage the hierarchical structure of distributed systems. Compared to SOTA memory-efficient solutions, it achieves a two-order-of-magnitude reduction in memory while maintaining comparable speed.
  arXiv  Detail & Related papers  (2024-10-22T17:59:30Z)
• Thinking Forward: Memory-Efficient Federated Finetuning of Language Models [21.438831528354513]
  Finetuning large language models (LLMs) in federated learning settings requires excessive memory for resource-constrained devices. In this paper, we introduce Spry, an FL algorithm that splits trainable weights of an LLM among participating clients. Spry achieves a low memory footprint, high accuracy, and fast convergence.
  arXiv  Detail & Related papers  (2024-05-24T13:37:48Z)
• AdaLomo: Low-memory Optimization with Adaptive Learning Rate [59.64965955386855]
  We introduce low-memory optimization with adaptive learning rate (AdaLomo) for large language models. AdaLomo results on par with AdamW, while significantly reducing memory requirements, thereby lowering the hardware barrier to training large language models.
  arXiv  Detail & Related papers  (2023-10-16T09:04:28Z)
• MAPLE-Edge: A Runtime Latency Predictor for Edge Devices [80.01591186546793]
  We propose MAPLE-Edge, an edge device-oriented extension of MAPLE, the state-of-the-art latency predictor for general purpose hardware. Compared to MAPLE, MAPLE-Edge can describe the runtime and target device platform using a much smaller set of CPU performance counters. We also demonstrate that unlike MAPLE which performs best when trained on a pool of devices sharing a common runtime, MAPLE-Edge can effectively generalize across runtimes.
  arXiv  Detail & Related papers  (2022-04-27T14:00:48Z)
• Puppeteer: A Random Forest-based Manager for Hardware Prefetchers across the Memory Hierarchy [1.5049442691806052]
  We propose Puppeteer as a hardware prefetcher manager that uses a suite of random forest regressors to determine at runtime which prefetcher should be ON at each level in the memory hierarchy. Using Puppeteer we improve IPC by 46.4% in 1 Core (1C), 25.8% in 4 Core (4C), and 11.9% in 8 Core (8C) processors on average across traces generated from SPEC 2017, SPEC2006, and Cloud suites with 10KB overhead.
  arXiv  Detail & Related papers  (2022-01-28T10:25:12Z)
• Edge AI without Compromise: Efficient, Versatile and Accurate Neurocomputing in Resistive Random-Access Memory [0.0]
  We present NeuRRAM - the first multimodal edge AI chip using RRAM CIM. We show record energy-efficiency $5times$ - $8times$ better than prior art across various computational bit-precisions. This work paves a way towards building highly efficient and reconfigurable edge AI hardware platforms.
  arXiv  Detail & Related papers  (2021-08-17T21:08:51Z)
• MAFAT: Memory-Aware Fusing and Tiling of Neural Networks for Accelerated Edge Inference [1.7894377200944507]
  Machine learning networks can easily exceed available memory, increasing latency due to excessive OS swapping. We propose a memory usage predictor coupled with a search algorithm to provide optimized fusing and tiling configurations. Results show that our approach can run in less than half the memory, and with a speedup of up to 2.78 under severe memory constraints.
  arXiv  Detail & Related papers  (2021-07-14T19:45:49Z)
• Non-Parametric Adaptive Network Pruning [125.4414216272874]
  We introduce non-parametric modeling to simplify the algorithm design. Inspired by the face recognition community, we use a message passing algorithm to obtain an adaptive number of exemplars. EPruner breaks the dependency on the training data in determining the "important" filters.
  arXiv  Detail & Related papers  (2021-01-20T06:18:38Z)
• A Privacy-Preserving-Oriented DNN Pruning and Mobile Acceleration Framework [56.57225686288006]
  Weight pruning of deep neural networks (DNNs) has been proposed to satisfy the limited storage and computing capability of mobile edge devices. Previous pruning methods mainly focus on reducing the model size and/or improving performance without considering the privacy of user data. We propose a privacy-preserving-oriented pruning and mobile acceleration framework that does not require the private training dataset.
  arXiv  Detail & Related papers  (2020-03-13T23:52:03Z)
• Joint Parameter-and-Bandwidth Allocation for Improving the Efficiency of Partitioned Edge Learning [73.82875010696849]
  Machine learning algorithms are deployed at the network edge for training artificial intelligence (AI) models. This paper focuses on the novel joint design of parameter (computation load) allocation and bandwidth allocation.
  arXiv  Detail & Related papers  (2020-03-10T05:52:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.