Fast Forward: Accelerating LLM Prefill with Predictive FFN Sparsity

• URL: http://arxiv.org/abs/2602.00397v1
• Date: Fri, 30 Jan 2026 23:23:40 GMT
• Title: Fast Forward: Accelerating LLM Prefill with Predictive FFN Sparsity
• Abstract summary: We introduce FastForward, a predictive sparsity framework that accelerates large language model (LLM) prefill through block-wise, context-aware FFN sparsity.<n>FastForward delivers up to 1.45$times$ compute-bound speedup at 50% FFN sparsity with $$ 6% accuracy loss compared to the dense baseline on LongBench.
• Score: 5.223181756178433
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The prefill stage of large language model (LLM) inference is a key computational bottleneck for long-context workloads. At short-to-moderate context lengths (1K--16K tokens), Feed-Forward Networks (FFNs) dominate this cost, accounting for most of the total FLOPs. Existing FFN sparsification methods, designed for autoregressive decoding, fail to exploit the prefill stage's parallelism and often degrade accuracy. To address this, we introduce FastForward, a predictive sparsity framework that accelerates LLM prefill through block-wise, context-aware FFN sparsity. FastForward combines (1) a lightweight expert predictor to select high-importance neurons per block, (2) an error compensation network to correct sparsity-induced errors, and (3) a layer-wise sparsity scheduler to allocate compute based on token-mixing importance. Across LLaMA and Qwen models up to 8B parameters, FastForward delivers up to 1.45$\times$ compute-bound speedup at 50% FFN sparsity with $<$ 6% accuracy loss compared to the dense baseline on LongBench, substantially reducing Time-to-First-Token (TTFT) for efficient, long-context LLM inference on constrained hardware.

Related papers

• Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding [51.48425605758328]
  We present Faster Flash Decoding (FFD), a novel hardware-algorithm co-design framework designed to break the memory wall in long-context decoding.<n>FFD integrates the selector and computer into a fully fused kernel, replacing external metadata indices with content-aware scanning.<n>It achieves up to 11.6x kernel-level speedup and scaling to 256K context length, with 2.37x end-to-end throughput improvement.
  arXiv  Detail & Related papers  (2026-08-31T15:13:20Z)
• Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMs [10.608163494702614]
  emphProx is a two-stage training-free framework for sparse SwiGLU FFNs.<n>It achieves up to a $1.99times$ end-to-end decoding speedup at 70% FFN sparsity.
  arXiv  Detail & Related papers  (2026-07-30T02:20:01Z)
• FastTPS: An Optimized Method for LLM Token Phase for AI accelerators [11.379739631959822]
  FastTPS is a high performance and low-precision loss method for accelerating the token-phase in LLM inference on general AI accelerators.<n>Our results confirm that FastTPS significantly alleviates memory bottlenecks in the token phase, delivering a 6x speed improvement.
  arXiv  Detail & Related papers  (2026-07-13T08:02:26Z)
• SVD Contextual Sparsity Predictors for Fast LLM Inference [0.0]
  This paper presents a framework for accelerating inference of ReGLU-based feed-forward networks (FFNs) within large language models (LLMs)<n>Experiments demonstrate up to a 1.8x reduction in end-to-end decoding time while maintaining less than 1% degradation in benchmark scores on tasks involving complex math and code generation.
  arXiv  Detail & Related papers  (2026-03-14T20:36:13Z)
• Theoretically Optimal Attention/FFN Ratios in Disaggregated LLM Serving [12.150964112168582]
  We develop a tractable analytical framework for sizing AFD bundles in a $r$A-$1$F topology.<n>We derive closed-form rules for the optimal A/F ratio that maximize average throughput per instance.<n>A trace-calibrated AFD simulator validates the theory.
  arXiv  Detail & Related papers  (2026-01-29T07:22:27Z)
• Fail Fast, Win Big: Rethinking the Drafting Strategy in Speculative Decoding via Diffusion LLMs [8.881949061263784]
  We show dLLM's speed from parallel decoding drastically lowers the risk of costly rejections.<n>We present FailFast, a dLLM-based speculative decoding framework that realizes this approach by dynamically adapting its speculation length.
  arXiv  Detail & Related papers  (2025-12-23T18:16:58Z)
• TeLLMe v2: An Efficient End-to-End Ternary LLM Prefill and Decode Accelerator with Table-Lookup Matmul on Edge FPGAs [9.646882213709814]
  TeLLMe is a table-lookup-based ternary LLM accelerator for low-power edge FPGAs.<n>It supports both prefill and autoregressive decoding using 1.58-bit weights and 8-bit activations.<n>Under a 5W power budget, TeLLMe delivers up to 25tokens/s decoding throughput.
  arXiv  Detail & Related papers  (2025-10-03T05:37:51Z)
• Fast-dLLM v2: Efficient Block-Diffusion LLM [64.38006546510337]
  Fast-dLLM v2 is a block diffusion language model that adapts pretrained AR models into dLLMs for parallel text generation.<n>This represents a 500x reduction in training data compared to full-attention diffusion LLMs such as Dream (580B tokens)
  arXiv  Detail & Related papers  (2025-09-30T14:40:18Z)
• Intra-request branch orchestration for efficient LLM reasoning [52.68946975865865]
  Large Language Models (LLMs) increasingly rely on inference-time reasoning algorithms to improve accuracy on complex tasks.<n>Prior work has largely focused on reducing token usage, often at the expense of accuracy, while overlooking other latency factors.<n>We present DUCHESS, an LLM serving system that reduces cost and latency without sacrificing accuracy through intra-request branch orchestration guided by predictions.
  arXiv  Detail & Related papers  (2025-09-29T15:52:08Z)
• R-Stitch: Dynamic Trajectory Stitching for Efficient Reasoning [80.104336426172]
  Chain-of-thought (CoT) enhances problem-solving ability of large language models.<n>CoT incurs substantial inference cost due to long autoregressive trajectories.<n>We introduce R-Stitch, a training-free hybrid decoding framework.
  arXiv  Detail & Related papers  (2025-07-23T08:14:36Z)
• BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity [66.94629945519125]
  We introduce a novel MoE architecture, BlockFFN, as well as its efficient training and deployment techniques.<n>Specifically, we use a router integrating ReLU activation and RMSNorm for differentiable and flexible routing.<n>Next, to promote both token-level sparsity (TLS) and chunk-level sparsity ( CLS), CLS-aware training objectives are designed, making BlockFFN more acceleration-friendly.
  arXiv  Detail & Related papers  (2025-07-11T17:28:56Z)
• SparseAccelerate: Efficient Long-Context Inference for Mid-Range GPUs [0.0]
  We introduce SparseAccelerate, a dynamic sparse attention method that adapts its sparsity patterns based on input characteristics.<n> Experimental results show that SparseAccelerate achieves up to a 1.04x reduction in Time-To-First-Token (TTTF) latency at 32K tokens.
  arXiv  Detail & Related papers  (2024-12-09T04:27:03Z)
• MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention [36.49445805074941]
  MInference (Milliontokens Inference) is a sparse calculation method designed to accelerate pre-filling of long-sequence processing. We demonstrate that MInference effectively reduces inference latency by up to 10x for pre-filling on an A100, while maintaining accuracy.
  arXiv  Detail & Related papers  (2024-07-02T17:59:56Z)
• FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping [49.66872823080736]
  Autoregressive Large Language Models (e.g., LLaMa, GPTs) are omnipresent achieving remarkable success in language understanding and generation. To mitigate overload incurred during generation, several early-exit and layer-dropping strategies have been proposed. We propose FFN-SkipLLM, which is an input-adaptive feed-forward skipping strategy.
  arXiv  Detail & Related papers  (2024-04-05T02:35:43Z)
• HiRE: High Recall Approximate Top-$k$ Estimation for Efficient LLM Inference [68.59839755875252]
  HiRE comprises of two novel components: (i) a compression scheme to cheaply predict top-$k$ rows/columns with high recall, followed by full computation restricted to the predicted subset, and (ii) DA-TOP-$k$: an efficient multi-device approximate top-$k$ operator. We demonstrate that on a one billion parameter model, HiRE applied to both the softmax as well as feedforward layers, achieves almost matching pretraining and downstream accuracy, and speeds up inference latency by $1.47times$ on a single TPUv5e device.
  arXiv  Detail & Related papers  (2024-02-14T18:04:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.