BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity

• URL: http://arxiv.org/abs/2507.08771v2
• Date: Wed, 30 Jul 2025 04:14:15 GMT
• Title: BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity
• Abstract summary: We introduce a novel MoE architecture, BlockFFN, as well as its efficient training and deployment techniques.<n>Specifically, we use a router integrating ReLU activation and RMSNorm for differentiable and flexible routing.<n>Next, to promote both token-level sparsity (TLS) and chunk-level sparsity ( CLS), CLS-aware training objectives are designed, making BlockFFN more acceleration-friendly.
• Score: 66.94629945519125
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: To alleviate the computational burden of large language models (LLMs), architectures with activation sparsity, represented by mixture-of-experts (MoE), have attracted increasing attention. However, the non-differentiable and inflexible routing of vanilla MoE hurts model performance. Moreover, while each token activates only a few parameters, these sparsely-activated architectures exhibit low chunk-level sparsity, indicating that the union of multiple consecutive tokens activates a large ratio of parameters. Such a sparsity pattern is unfriendly for acceleration under low-resource conditions (e.g., end-side devices) and incompatible with mainstream acceleration techniques (e.g., speculative decoding). To address these challenges, we introduce a novel MoE architecture, BlockFFN, as well as its efficient training and deployment techniques. Specifically, we use a router integrating ReLU activation and RMSNorm for differentiable and flexible routing. Next, to promote both token-level sparsity (TLS) and chunk-level sparsity (CLS), CLS-aware training objectives are designed, making BlockFFN more acceleration-friendly. Finally, we implement efficient acceleration kernels, combining activation sparsity and speculative decoding for the first time. The experimental results demonstrate the superior performance of BlockFFN over other MoE baselines, achieving over 80% TLS and 70% 8-token CLS. Our kernels achieve up to 3.67$\times$ speedup on real end-side devices than dense models. All codes and checkpoints are available publicly (https://github.com/thunlp/BlockFFN).

Related papers

• Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding [51.48425605758328]
  We present Faster Flash Decoding (FFD), a novel hardware-algorithm co-design framework designed to break the memory wall in long-context decoding.<n>FFD integrates the selector and computer into a fully fused kernel, replacing external metadata indices with content-aware scanning.<n>It achieves up to 11.6x kernel-level speedup and scaling to 256K context length, with 2.37x end-to-end throughput improvement.
  arXiv  Detail & Related papers  (2026-08-31T15:13:20Z)
• Empowering Compact LLMs with Fusion of Layer-wise Exits for Recommendation [59.216721553953]
  Large language model-based recommender systems (LLM-RSs) have demonstrated remarkable capabilities, but are computationally unsustainable for many real-world applications.<n>We propose a discriminative framework that enhances compact LLMs while retaining scalable full-corpus ranking.<n>Experiments on three real-world datasets with Qwen 3 1.7B and Llama 3.2 3B show that FLEXRec achieves state-of-the-art accuracy while remaining highly efficient.
  arXiv  Detail & Related papers  (2026-08-18T03:14:28Z)
• FluxBin: Flexible LUT-based Ultra-low-bit LLM Inference by Algorithm-Kernel Synergy [32.21687694159134]
  FluxBin is an algorithm that synerizes binary quantization with a highly optimized kernel.<n>FluxBin achieves up to $5.92times speedup and $10.19times energy savings.<n>This effectively enables the deployment of 70B-scale models on one single A100 bases with a $4times memory reduction.
  arXiv  Detail & Related papers  (2026-08-16T08:01:06Z)
• AdaFuse: Accelerating Dynamic Adapter Inference via Token-Level Pre-Gating and Fused Kernel Optimization [84.25316984309725]
  Integration of dynamic, sparse structures with parameter-efficient adapters (e.g., LoRA) is a powerful technique for enhancing Large Language Models (LLMs)<n>Despite minimal increases in computational load, the computation latency often skyrockets, leading to decoding speeds slowing by over 2.5 times.<n>AdaFuse is a framework built on a tight co-design between the algorithm and the underlying hardware system to enable efficient dynamic adapter execution.
  arXiv  Detail & Related papers  (2026-03-12T12:46:42Z)
• Fast-dLLM v2: Efficient Block-Diffusion LLM [64.38006546510337]
  Fast-dLLM v2 is a block diffusion language model that adapts pretrained AR models into dLLMs for parallel text generation.<n>This represents a 500x reduction in training data compared to full-attention diffusion LLMs such as Dream (580B tokens)
  arXiv  Detail & Related papers  (2025-09-30T14:40:18Z)
• FlashOmni: A Unified Sparse Attention Engine for Diffusion Transformers [7.026182341295719]
  Flash Omni is a unified sparse attention engine compatible with arbitrary DiT architectures.<n>It delivers near-linear, closely matching the sparsity ratio speedup in attention and GEMM-$Q$, and achieves 2.5$times$-3.8$times$ acceleration in GEMM-$O$.
  arXiv  Detail & Related papers  (2025-09-29T18:57:14Z)
• Spark Transformer: Reactivating Sparsity in FFN and Attention [63.20677098823873]
  We introduce Spark Transformer, a novel architecture that achieves a high level of activation sparsity in both FFN and the attention mechanism.<n>This sparsity translates to a 2.5x reduction in FLOPs, leading to decoding wall-time speedups of up to 1.79x on CPU and 1.40x on GPU.
  arXiv  Detail & Related papers  (2025-06-07T03:51:13Z)
• CMoE: Converting Mixture-of-Experts from Dense to Accelerate LLM Inference [33.871080938643566]
  We present CMoE, a framework that rapidly transforms dense language models into mixture-of-experts (MoEs) without training.<n>Experiments demonstrate that, with activation ratio of 75%, it achieves remarkable results in terms of perplexity.<n>A CMoE configuration activating just 25% of parameters reduces end-to-end latency by 1.5x while preserving usable perplexity without additional training.
  arXiv  Detail & Related papers  (2025-02-06T14:05:30Z)
• Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters [20.093224415258174]
  Activation sparsity is determined by activation functions, and commonly used ones like SwiGLU and GeGLU exhibit limited sparsity. We propose a novel dReLU function, which is designed to improve LLM activation sparsity, along with a high-quality training data mixture ratio. On mobile phones, our TurboSparse-Mixtral-47B achieves an inference speed of 11 tokens per second.
  arXiv  Detail & Related papers  (2024-06-10T01:21:59Z)
• Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generation [31.657608562937543]
  We introduce GRIFFIN, a training-free and calibration-free method that selects unique FF experts at the sequence level for efficient generation. GRIFFIN maintains the original model's performance with little to no degradation on a variety of classification and generation tasks.
  arXiv  Detail & Related papers  (2024-04-01T17:56:06Z)
• FFSplit: Split Feed-Forward Network For Optimizing Accuracy-Efficiency Trade-off in Language Model Inference [57.119047493787185]
  This paper shows how to reduce model size by 43.1% and bring $1.25sim1.56times$ wall clock time speedup on different hardware with negligible accuracy drop. In practice, our method can reduce model size by 43.1% and bring $1.25sim1.56times$ wall clock time speedup on different hardware with negligible accuracy drop.
  arXiv  Detail & Related papers  (2024-01-08T17:29:16Z)
• Efficient Few-Shot Object Detection via Knowledge Inheritance [62.36414544915032]
  Few-shot object detection (FSOD) aims at learning a generic detector that can adapt to unseen tasks with scarce training samples. We present an efficient pretrain-transfer framework (PTF) baseline with no computational increment. We also propose an adaptive length re-scaling (ALR) strategy to alleviate the vector length inconsistency between the predicted novel weights and the pretrained base weights.
  arXiv  Detail & Related papers  (2022-03-23T06:24:31Z)
• FastFlowNet: A Lightweight Network for Fast Optical Flow Estimation [81.76975488010213]
  Dense optical flow estimation plays a key role in many robotic vision tasks. Current networks often occupy large number of parameters and require heavy computation costs. Our proposed FastFlowNet works in the well-known coarse-to-fine manner with following innovations.
  arXiv  Detail & Related papers  (2021-03-08T03:09:37Z)
• Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch [75.69506249886622]
  Sparsity in Deep Neural Networks (DNNs) has been widely studied to compress and accelerate the models on resource-constrained environments. In this paper, we are the first to study training from scratch an N:M fine-grained structured sparse network.
  arXiv  Detail & Related papers  (2021-02-08T05:55:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.