CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration

• URL: http://arxiv.org/abs/2605.11186v1
• Date: Mon, 11 May 2026 19:50:08 GMT
• Title: CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration
• Abstract summary: Auto-regressive decoding in Large Language Models (LLMs) is inherently memory-bound.<n>We propose CATS, a self-speculative decoding framework that conducts cascaded verification and correction on memory-limited devices.<n> CATS can achieve a wall-clock speedup of up to 5.08x with no degradation in generation quality, outperforming the SOTA method by up to 1.45x under edge memory constraints.
• Score: 2.6196419303092466
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Auto-regressive decoding in Large Language Models (LLMs) is inherently memory-bound: every generation step requires loading the model weights and intermediate results from memory (e.g., High-Bandwidth Memory (HBM) for GPU servers), making throughput bottlenecked by memory bandwidth rather than compute. Speculative decoding addresses this by enabling parallel verification of multiple draft tokens, effectively amortizing the cost of each target-model call. However, existing speculative decoding methods are designed under the assumption that HBM is sufficiently large to hold both the target model and an auxiliary draft model simultaneously -- an assumption that breaks down on memory-constrained devices such as edge platforms with limited DRAM. We analyze the inference bottleneck in this memory-limited regime and propose CATS, a self-speculative decoding framework that conducts cascaded verification and correction based on the memory budget and parameter offloading patterns on memory-limited devices. This design maximizes token acceptance rate and end-to-end speedup while keeping the peak memory footprint on the device equal to that of the target model alone. We evaluate CATS on different models across five benchmarks on real edge devices. CATS can achieve a wall-clock speedup of up to 5.08x with no degradation in generation quality, outperforming the SOTA method by up to 1.45x under edge memory constraints.

Related papers

• LLM Inference in a Flash! [66.43896490006965]
  Large Language Models (LLMs) have shown impressive capabilities across a range of natural language processing tasks.<n>We present an end-to-end integer-only quantization approach to eliminate expensive floating-point computations.
  arXiv  Detail & Related papers  (2026-09-14T18:04:30Z)
• Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory [65.39827296429527]
  Scaling conditional memory offers a promising way to increase language-model capacity.<n>Existing methods such as Engram learn large memory tables from scratch during pre-training.<n>We propose Memory Grafting, a conditional memory scaling method that utilizes frozen hidden states from a grafting model as conditional n-gram memory.
  arXiv  Detail & Related papers  (2026-05-20T09:35:04Z)
• Beyond N-gram: Data-Aware X-GRAM Extraction for Efficient Embedding Parameter Scaling [24.48256429931821]
  X-GRAM is a frequency-aware dynamic token-injection framework.<n>It employs hybrid hashing and alias mixing to compress the tail while preserving head capacity.<n>It refines retrieved vectors via normalized SwiGLU ShortConv to extract diverse local n-gram features.
  arXiv  Detail & Related papers  (2026-04-23T14:27:10Z)
• Parameter Efficiency Is Not Memory Efficiency: Rethinking Fine-Tuning for On-Device LLM Adaptation [1.2559585990041289]
  LARS (Low-memory Activation-Rank Subspace) is a novel adaptation framework that decouples memory consumption from sequence length.<n>We show that LARS reduces the memory footprint by an average of 33.54% on GPUs and 51.95% on CPUs in comparison to LoRA across reasoning, understanding and long-context datasets.
  arXiv  Detail & Related papers  (2026-04-03T17:05:33Z)
• MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens [38.762431789628074]
  We present Memory Sparse Attention (MSA), an end-to-end trainable, efficient, and massively scalable memory model framework.<n>MSA achieves linear complexity in both training and inference while maintaining exceptional stability.<n>We also propose Memory Interleaving to facilitate complex multi-hop reasoning across scattered memory segments.
  arXiv  Detail & Related papers  (2026-03-06T02:29:54Z)
• NextMem: Towards Latent Factual Memory for LLM-based Agents [58.35585202907478]
  NextMem is a latent factual memory framework that utilizes an autoregressive autoencoder to efficiently construct latent memory.<n>Extensive experiments demonstrate that NextMem achieves superior performance.
  arXiv  Detail & Related papers  (2026-02-26T14:35:27Z)
• QMC: Efficient SLM Edge Inference via Outlier-Aware Quantization and Emergent Memories Co-Design [8.787715061109163]
  Outlier-aware Quantization Memory Co-design (QMC) is a retraining-free quantization with a novel heterogeneous memory architecture.<n>QMC reduces memory usage by 6.3x-7.3x, external data transfers by 7.6x, energy by 11.7x, and latency by 12.5x when compared to FP16.
  arXiv  Detail & Related papers  (2026-01-21T00:11:34Z)
• Memory- and Latency-Constrained Inference of Large Language Models via Adaptive Split Computing [8.705453442427585]
  Large language models (LLMs) have achieved near-human performance across diverse reasoning tasks.<n>Their deployment on resource-constrained Internet-of-Things (IoT) devices remains impractical due to massive parameter footprints and memory-intensive autoregressive decoding.<n>This work introduces the first autoregressive-aware split computing framework designed explicitly for LLM deployment on edge devices.
  arXiv  Detail & Related papers  (2025-11-06T02:55:07Z)
• Lizard: An Efficient Linearization Framework for Large Language Models [113.87302474262798]
  We propose Lizard, a linearization framework that transforms pretrained Transformer-based Large Language Models (LLMs) into subquadratic architectures.<n>Lizard addresses these limitations by introducing a subquadratic attention mechanism that closely approximates softmax attention while preserving model quality.<n>We show that Lizard achieves near-lossless recovery of its teacher model's performance, significantly outperforming previous methods by up to 9.4 - 24.5 points on the 5-shot MMLU benchmark.
  arXiv  Detail & Related papers  (2025-07-11T21:19:18Z)
• QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache [67.84112700032007]
  Large Language Models (LLMs) are increasingly being deployed on edge devices for long-context settings.<n>In these scenarios, the Key-Value ( KV) cache is the primary bottleneck in terms of both GPU memory and latency.<n>We propose a novel self-speculative decoding framework, QuantSpec, where the draft model shares the architecture of the target model but employs a hierarchical 4-bit quantized KV cache and 4-bit quantized weights for acceleration.
  arXiv  Detail & Related papers  (2025-02-05T20:43:48Z)
• LiVOS: Light Video Object Segmentation with Gated Linear Matching [116.58237547253935]
  LiVOS is a lightweight memory network that employs linear matching via linear attention. For longer and higher-resolution videos, it matched STM-based methods with 53% less GPU memory and supports 4096p inference on a 32G consumer-grade GPU.
  arXiv  Detail & Related papers  (2024-11-05T05:36:17Z)
• Memformer: A Memory-Augmented Transformer for Sequence Modeling [55.780849185884996]
  We present Memformer, an efficient neural network for sequence modeling. Our model achieves linear time complexity and constant memory space complexity when processing long sequences.
  arXiv  Detail & Related papers  (2020-10-14T09:03:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.