LayerRoute: Adaptive Layer-Skipping with LoRA-Preserved Quality for Efficient LLM Inference

• URL: http://arxiv.org/abs/2609.13682v1
• Date: Sat, 12 Sep 2026 03:30:59 GMT
• Title: LayerRoute: Adaptive Layer-Skipping with LoRA-Preserved Quality for Efficient LLM Inference
• Abstract summary: LayerRoute is a parameter-efficient method for adaptive transformer layer-skipping.<n>It combines per-layer hard-gated routing with joint LoRA fine-tuning.<n>It trains in under 7 minutes on a single A100 and adds negligible overhead.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce LayerRoute, a parameter-efficient method for adaptive transformer layer-skipping that combines per-layer hard-gated routing (trained via a straight-through estimator) with joint LoRA fine-tuning. LayerRoute augments each of the 24 transformer blocks in Qwen2.5-0.5B-Instruct with a lightweight per-layer router (~21.5K parameters) and LoRA adapters (rank 8, ~1.08M parameters), training both jointly under a gate-regularized language-modeling objective. Across 10 independently-seeded training runs, LayerRoute converges to an identical skip-pattern structure in every run - a consistent set of 9 middle layers (8-16) becomes skip-eligible in all 10 seeds - and delivers genuine, verified wallclock speedup in every run (1.02x-1.06x, mean 1.04x). Quality is preserved or improved in every configuration tested: joint LoRA adaptation yields a perplexity improvement over the unmodified backbone in all 10 seeds (mean delta = -1.16 and -1.11 across the two evaluation splits used). We further verify the router performs genuine, non-trivial per-input computation: gate decisions in skip-eligible layers change the actual skip/run outcome for 87-100% of held-out samples, confirming real input-dependent routing rather than a fixed pruning pattern. LayerRoute trains in under 7 minutes on a single A100 and adds negligible overhead beyond the routing decision itself. We report our full reproducibility methodology, including a systematic diagnostic investigation into what determines the router's per-input decisions, as part of this work.

Related papers

• Do Dynamic Routers Need Memory? HeRo: History-Aware Routing for Efficient LLM Inference [13.935681015472262]
  Dynamic layer routing reduces the inference cost of Large Language Models (LLMs) by learning to skip layers for individual tokens.<n>Existing methods treat each routing decision as a local operation conditioned solely on the current hidden state.<n>We propose History-Aware Routing (HeRo), a dynamic routing framework that maintains an explicit routing state across model depth.
  arXiv  Detail & Related papers  (2026-09-08T03:20:07Z)
• Disagree to Explore, Agree to Commit: Routing-Guided Test-Time Scaling for Software Agents [60.650808962786364]
  We study whether native MoE router traces can guide steering and selection without an external judge or selection-time test execution.<n>Our analysis shows that routing provides a robust behavioral role signal; token-granular readouts and decision-matched comparison sets turn it into effective control.<n>We evaluate on SWE-bench Verified using open-weight sparse MoE agents across scales and reasoning-effort settings.
  arXiv  Detail & Related papers  (2026-08-23T03:07:03Z)
• Agent-as-a-Router: Agentic Model Routing for Coding Tasks [54.00153674547733]
  Agent-as-a-router (AC) formalizes routing as a C-A-F loop (Context->Action->FeedbackContext)<n>AC achieves the lowest cumulative regret on in-distribution tasks and generalizes to out-of-distribution agentic-programming tasks.
  arXiv  Detail & Related papers  (2026-06-22T06:37:31Z)
• LayerRoute: Input-Conditioned Adaptive Layer Skipping via LoRA Fine-Tuning for Agentic Language Models [0.0]
  We introduce LayerRoute, a lightweight adapter that learns to selectively skip transformer blocks on a per-input basis.<n> LayerRoute augments each of the 24 transformer blocks in Qwen2.5-0.5B-Instruct with: (1) a per-layer router that outputs a hard binary gate via the straight-through estimator, and (2) LoRA adapters (rank 8, 1.08M parameters) on the Q/K/V/O attention projections.<n>A single end-to-end training pass on agentic data (Hermes, Glaive, GSM8K, Turing) with a gate regularisation term
  arXiv  Detail & Related papers  (2026-06-01T07:50:05Z)
• Learning Agent Routing From Early Experience [55.88886987958933]
  Boundary is a training-free routing framework that uses early behavioral experience and rubric-guided reasoning to decide whether to answer a query with direct inference or escalate to an agent.<n>We show that Boundary reduces inference time by 60.6% compared to the agent while improving performance by 28.6% over direct LLM inference.
  arXiv  Detail & Related papers  (2026-05-08T03:18:40Z)
• Aletheia: Gradient-Guided Layer Selection for Efficient LoRA Fine-Tuning Across Architectures [0.0]
  Low-Rank Adaptation (LoRA) has become the dominant parameter-efficient fine-tuning method for large models.<n>We introduce Aletheia, a gradient-guided layer selection method.<n>We show that Aletheia can make LoRA fine-tuning materially more efficient without introducing major downstream damage on the evaluated set.
  arXiv  Detail & Related papers  (2026-04-04T10:24:12Z)
• Task-Conditioned Routing Signatures in Sparse Mixture-of-Experts Transformers [0.0]
  We introduce routing signatures, a vector representation summarizing expert activation patterns across layers for a given prompt.<n>We show that prompts from the same task category induce highly similar routing signatures, while prompts from different categories exhibit substantially lower similarity.<n>A logistic regression trained solely on routing signatures achieves 92.5% +/- 6.1% cross-validated accuracy on four-way task classification.
  arXiv  Detail & Related papers  (2026-03-11T12:45:53Z)
• Dr.LLM: Dynamic Layer Routing in LLMs [55.11953638340419]
  Dr.LLM is a retrofittable framework that equips pretrained models with lightweight per-layer routers deciding to skip, execute, or repeat a block.<n>On ARC (logic) and DART (math), Dr.LLM improves accuracy by up to +3.4%p while saving 5 layers per example on average.
  arXiv  Detail & Related papers  (2025-10-14T17:51:26Z)
• Semi-DETR: Semi-Supervised Object Detection with Detection Transformers [105.45018934087076]
  We analyze the DETR-based framework on semi-supervised object detection (SSOD) We present Semi-DETR, the first transformer-based end-to-end semi-supervised object detector. Our method outperforms all state-of-the-art methods by clear margins.
  arXiv  Detail & Related papers  (2023-07-16T16:32:14Z)
• End-to-End Sensitivity-Based Filter Pruning [49.61707925611295]
  We present a sensitivity-based filter pruning algorithm (SbF-Pruner) to learn the importance scores of filters of each layer end-to-end. Our method learns the scores from the filter weights, enabling it to account for the correlations between the filters of each layer.
  arXiv  Detail & Related papers  (2022-04-15T10:21:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.