AdaFuse: Accelerating Dynamic Adapter Inference via Token-Level Pre-Gating and Fused Kernel Optimization

• URL: http://arxiv.org/abs/2603.11873v1
• Date: Thu, 12 Mar 2026 12:46:42 GMT
• Title: AdaFuse: Accelerating Dynamic Adapter Inference via Token-Level Pre-Gating and Fused Kernel Optimization
• Abstract summary: Integration of dynamic, sparse structures with parameter-efficient adapters (e.g., LoRA) is a powerful technique for enhancing Large Language Models (LLMs)<n>Despite minimal increases in computational load, the computation latency often skyrockets, leading to decoding speeds slowing by over 2.5 times.<n>AdaFuse is a framework built on a tight co-design between the algorithm and the underlying hardware system to enable efficient dynamic adapter execution.
• Score: 84.25316984309725
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The integration of dynamic, sparse structures like Mixture-of-Experts (MoE) with parameter-efficient adapters (e.g., LoRA) is a powerful technique for enhancing Large Language Models (LLMs). However, this architectural enhancement comes at a steep cost: despite minimal increases in computational load, the inference latency often skyrockets, leading to decoding speeds slowing by over 2.5 times. Through a fine-grained performance analysis, we pinpoint the primary bottleneck not in the computation itself, but in the severe overhead from fragmented, sequential CUDA kernel launches required for conventional dynamic routing. To address this challenge, we introduce AdaFuse, a framework built on a tight co-design between the algorithm and the underlying hardware system to enable efficient dynamic adapter execution. Departing from conventional layer-wise or block-wise routing, AdaFuse employs a token-level pre-gating strategy, which makes a single, global routing decision for all adapter layers before a token is processed. This "decide-once, apply-everywhere" approach effectively staticizes the execution path for each token, creating an opportunity for holistic optimization. We capitalize on this by developing a custom CUDA kernel that performs a fused switching operation, merging the parameters of all selected LoRA adapters into the backbone model in a single, efficient pass. Experimental results on popular open-source LLMs show that AdaFuse achieves accuracy on par with state-of-the-art dynamic adapters while drastically cutting decoding latency by a factor of over 2.4x, thereby bridging the gap between model capability and inference efficiency.

Related papers

• Accelerating Unified Multimodal Models with Core-Expansion Routing and Unified Computation Scheduling [61.28599393707943]
  Unified multimodal models jointly support understanding and generation, but incur substantial redundant computation across tokens, layers, and generation timesteps.<n>We propose a task-shared core scorer and progress-conditioned generation expansions, optimized through generation decomposition and cross-task core alignment.<n> Experiments on two representative UMM architectures demonstrate consistent quality--efficiency improvements across both tasks, retaining 98.03% of dense understanding performance with a 1.93times$ end-to-end inference speedup.
  arXiv  Detail & Related papers  (2026-08-29T14:27:55Z)
• WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning [12.767703956755751]
  We present the first end-to-end differentiable token-level dynamic width pruning framework for both prefill and decode scenarios.<n>We propose a pruning- kernel co-design framework that decomposes dynamic sparsity acceleration into mask reordering, hardware-agnostic block-level skipping, and hardware-dependent intra-block skipping.
  arXiv  Detail & Related papers  (2026-07-30T16:01:03Z)
• Dynamic-dLLM: Dynamic Cache-Budget and Adaptive Parallel Decoding for Training-Free Acceleration of Diffusion LLM [35.77320928871856]
  We propose Dynamic-dLLM, a training-free framework that enhances dLLM inference efficiency through two components.<n>Experiments on models like LLaDA-8B-Instruct, LLaDA-1.5, and Dream-v0-7B-Instruct demonstrate that Dynamic-dLLM significantly improves inference speed.
  arXiv  Detail & Related papers  (2026-05-27T02:47:50Z)
• Auto-FlexSwitch: Efficient Dynamic Model Merging via Learnable Task Vector Compression [54.12624130407679]
  We propose T-Switch, which decomposes task vectors into three compact components.<n>We then introduce Auto-Switch, a training-free merging scheme that automatically composes task vectors.<n>We present Auto-FlexSwitch, a dynamic model merging approach that supports highly efficient task vector compression.
  arXiv  Detail & Related papers  (2026-04-30T16:58:05Z)
• AutoQRA: Joint Optimization of Mixed-Precision Quantization and Low-rank Adapters for Efficient LLM Fine-Tuning [23.59600455731982]
  We propose a joint optimization framework that simultaneously optimize the bit-width and LoRA rank configuration for each layer during the mixed quantized fine-tuning process.<n>Experiments show that AutoQRA achieves performance close to full-precision fine-tuning with a memory footprint comparable to uniform 4-bit methods.
  arXiv  Detail & Related papers  (2026-02-25T07:18:08Z)
• Rethinking Autoregressive Models for Lossless Image Compression via Hierarchical Parallelism and Progressive Adaptation [75.58269386927076]
  Autoregressive (AR) models are often dismissed as impractical due to prohibitive computational cost.<n>This work re-thinks this paradigm, introducing a framework built on hierarchical parallelism and progressive adaptation.<n> Experiments on diverse datasets (natural, satellite, medical) validate that our method achieves new state-of-the-art compression.
  arXiv  Detail & Related papers  (2025-11-14T06:27:58Z)
• Eliminating Multi-GPU Performance Taxes: A Systems Approach to Efficient Distributed LLMs [61.953548065938385]
  We introduce the ''Three Taxes'' (Bulk Synchronous, Inter- Kernel Data Locality, and Kernel Launch Overhead) as an analytical framework.<n>We propose moving beyond the rigid BSP model to address key inefficiencies in distributed GPU execution.<n>We observe a 10-20% speedup in end-to-end latency over BSP-based approaches.
  arXiv  Detail & Related papers  (2025-11-04T01:15:44Z)
• HALO: Memory-Centric Heterogeneous Accelerator with 2.5D Integration for Low-Batch LLM Inference [8.057006406834462]
  Large Language Models (LLMs) have driven a growing demand for efficient inference in latency-sensitive applications.<n>We present HALO, a heterogeneous memory-centric accelerator for these challenges.<n>We show that HALO achieves up to 18x geometric mean speedup over AttAcc, an attention-optimized mapping and 2.5x over CENT.
  arXiv  Detail & Related papers  (2025-10-03T02:20:17Z)
• Inter2Former: Dynamic Hybrid Attention for Efficient High-Precision Interactive [58.0729162588429]
  Interactive segmentation improves annotation efficiency by segmenting target regions from user prompts.<n>Current approaches face a critical trade-off: dense-token methods achieve superior accuracy but suffer from prohibitively slow processing on CPU devices.<n>We propose Inter2Former to address this challenge by optimizing computation allocation in dense-token processing.
  arXiv  Detail & Related papers  (2025-07-13T12:33:37Z)
• FiRST: Finetuning Router-Selective Transformers for Input-Adaptive Latency Reduction [16.84400858871298]
  We propose FiRST, an algorithm that reduces latency by using layer-specific routers to select a subset of transformer layers adaptively for each input sequence.<n>FiRST preserves compatibility with KV caching enabling faster inference while being quality-aware.<n>Our approach reveals that input adaptivity is critical - indeed, different task-specific middle layers play a crucial role in evolving hidden representations depending on tasks.
  arXiv  Detail & Related papers  (2024-10-16T12:45:35Z)
• LoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-design [23.874726096958135]
  We analyze the fine-grained costs of the dynamic adapters and find that the fragmented kernel calls are the root cause. Unlike most existing dynamic structures that adopt layer-wise or block-wise dynamic routing, LoRA-Switch introduces a token-wise routing mechanism. For efficiency, this switching is implemented with an optimized kernel, which fuses the operations for all LoRA adapters at once.
  arXiv  Detail & Related papers  (2024-05-28T01:53:26Z)
• An Adaptive Device-Edge Co-Inference Framework Based on Soft Actor-Critic [72.35307086274912]
  High-dimension parameter model and large-scale mathematical calculation restrict execution efficiency, especially for Internet of Things (IoT) devices. We propose a new Deep Reinforcement Learning (DRL)-Soft Actor Critic for discrete (SAC-d), which generates the emphexit point, emphexit point, and emphcompressing bits by soft policy iterations. Based on the latency and accuracy aware reward design, such an computation can well adapt to the complex environment like dynamic wireless channel and arbitrary processing, and is capable of supporting the 5G URL
  arXiv  Detail & Related papers  (2022-01-09T09:31:50Z)
• Learning to Generate Content-Aware Dynamic Detectors [62.74209921174237]
  We introduce a newpective of designing efficient detectors, which is automatically generating sample-adaptive model architecture. We introduce a course-to-fine strat-egy tailored for object detection to guide the learning of dynamic routing. Experiments on MS-COCO dataset demonstrate that CADDet achieves 1.8 higher mAP with 10% fewer FLOPs compared with vanilla routing.
  arXiv  Detail & Related papers  (2020-12-08T08:05:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.