LayerRoute: Input-Conditioned Adaptive Layer Skipping via LoRA Fine-Tuning for Agentic Language Models

• URL: http://arxiv.org/abs/2606.01838v1
• Date: Mon, 01 Jun 2026 07:50:05 GMT
• Title: LayerRoute: Input-Conditioned Adaptive Layer Skipping via LoRA Fine-Tuning for Agentic Language Models
• Abstract summary: We introduce LayerRoute, a lightweight adapter that learns to selectively skip transformer blocks on a per-input basis.<n> LayerRoute augments each of the 24 transformer blocks in Qwen2.5-0.5B-Instruct with: (1) a per-layer router that outputs a hard binary gate via the straight-through estimator, and (2) LoRA adapters (rank 8, 1.08M parameters) on the Q/K/V/O attention projections.<n>A single end-to-end training pass on agentic data (Hermes, Glaive, GSM8K, Turing) with a gate regularisation term
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic language model systems alternate between two structurally distinct step types: structured tool calls (short, deterministic, low perplexity) and open-ended planning/reasoning steps (long, complex, high perplexity). Despite this heterogeneity, current inference systems apply identical compute to every step. We introduce LayerRoute, a lightweight adapter that learns to selectively skip transformer blocks on a per-input basis. LayerRoute augments each of the 24 transformer blocks in Qwen2.5-0.5B-Instruct with: (1) a per-layer router (~897 parameters, Linear(896,1)) that outputs a hard binary gate via the straight-through estimator, and (2) LoRA adapters (rank 8, ~1.08M parameters) on the Q/K/V/O attention projections. The backbone weights remain frozen. A single end-to-end training pass on agentic data (Hermes, Glaive, GSM8K, Turing) with a gate regularisation term forces the system to discover which blocks are skippable per input type. After 3,000 steps (6.4 minutes on an A100 40GB), LayerRoute achieves a 12.91% skip differential: tool calls skip 15.25% of FLOPs while planning steps skip only 2.34%, using only 1.10M trainable parameters (0.22% of the 494M backbone). Quality improves over the base model due to LoRA adaptation, with perplexity delta of -1.29 on tool calls and -1.30 on planning.

Related papers

• LayerRoute: Adaptive Layer-Skipping with LoRA-Preserved Quality for Efficient LLM Inference [0.0]
  LayerRoute is a parameter-efficient method for adaptive transformer layer-skipping.<n>It combines per-layer hard-gated routing with joint LoRA fine-tuning.<n>It trains in under 7 minutes on a single A100 and adds negligible overhead.
  arXiv  Detail & Related papers  (2026-09-12T03:30:59Z)
• Do Dynamic Routers Need Memory? HeRo: History-Aware Routing for Efficient LLM Inference [13.935681015472262]
  Dynamic layer routing reduces the inference cost of Large Language Models (LLMs) by learning to skip layers for individual tokens.<n>Existing methods treat each routing decision as a local operation conditioned solely on the current hidden state.<n>We propose History-Aware Routing (HeRo), a dynamic routing framework that maintains an explicit routing state across model depth.
  arXiv  Detail & Related papers  (2026-09-08T03:20:07Z)
• CRISP: Cliff-awaRe Input-adaptive Sparse Prefilling with Structural-Mass-Motivated Routing [90.63350360476294]
  We present CRISP (Cliff-awaRe Input-adaptive Sparse Prefilling), which identifies and addresses two structural challenges in this dynamic routing paradigm.<n>We show that the routing decision can be read directly off the structure of the proxy attention map.<n>We formalize the post-softmax mass cliff and demonstrate theoretically that strictly cumulative coverage thresholds accumulate O(n) background noise at long contexts.
  arXiv  Detail & Related papers  (2026-09-01T22:49:46Z)
• Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality [0.0]
  We build a sparse Mixture-of-Experts (MoE) architecture using cosine-similarity routing against learned centroids.<n>We find that routing topology does not determine perplexity (PPL)<n>Expert-level and causal controllability -- which coexist with topology-level equifinality -- are explored in a companion paper.
  arXiv  Detail & Related papers  (2026-04-15T21:02:02Z)
• When Does Content-Based Routing Work? Representation Requirements for Selective Attention in Hybrid Sequence Models [0.0]
  We identify a routing paradox in hybrid recurrent-attention architectures.<n>We show that content-based routing requires exactly the pairwise computation that routing is designed to avoid.
  arXiv  Detail & Related papers  (2026-03-22T01:04:57Z)
• Directional Routing in Transformers [0.0]
  directional routing is a lightweight mechanism that gives each transformer attention head learned suppression directions controlled by a shared router.<n>We trace the resulting circuits through mechanistic interpretability.<n> routing reduces perplexity 31-56% relative to the baseline, though downstream multiple-choice benchmarks do not yet reflect these gains.
  arXiv  Detail & Related papers  (2026-03-16T07:28:22Z)
• The Discrete Charm of the MLP: Binary Routing of Continuous Signals in Transformer Feed-Forward Layers [0.0]
  We show that layers in language models perform binary routing of continuous signals.<n>We find that specific neurons implement a consensus architecture that are 93-98% mutually exclusive.<n>We propose that the well-established piecewise-affine characterization of deep networks can be complemented by a routing characterization.
  arXiv  Detail & Related papers  (2026-03-11T17:14:57Z)
• Dr.LLM: Dynamic Layer Routing in LLMs [55.11953638340419]
  Dr.LLM is a retrofittable framework that equips pretrained models with lightweight per-layer routers deciding to skip, execute, or repeat a block.<n>On ARC (logic) and DART (math), Dr.LLM improves accuracy by up to +3.4%p while saving 5 layers per example on average.
  arXiv  Detail & Related papers  (2025-10-14T17:51:26Z)
• Skip2-LoRA: A Lightweight On-device DNN Fine-tuning Method for Low-cost Edge Devices [7.219286228148705]
  This paper proposes Skip2-LoRA as a lightweight fine-tuning method for deep neural networks. In our approach, trainable LoRA adapters are inserted between the last layer and every other layer to enhance the network expressive power. Our results show that Skip2-LoRA reduces the fine-tuning time by 90.0% on average compared to the counterpart that has the same number of trainable parameters.
  arXiv  Detail & Related papers  (2024-10-28T14:35:12Z)
• Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching [56.286064975443026]
  We make an interesting and somehow surprising observation: the computation of a large proportion of layers in the diffusion transformer, through a caching mechanism, can be readily removed even without updating the model parameters. We introduce a novel scheme, named Learningto-Cache (L2C), that learns to conduct caching in a dynamic manner for diffusion transformers. Experimental results show that L2C largely outperforms samplers such as DDIM and DPM-r, alongside prior cache-based methods at the same inference speed.
  arXiv  Detail & Related papers  (2024-06-03T18:49:57Z)
• Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performance [87.19164603145056]
  We propose LoRAT, a method that unveils the power of large ViT model for tracking within laboratory-level resources. The essence of our work lies in adapting LoRA, a technique that fine-tunes a small subset of model parameters without adding inference latency. We design an anchor-free head solely based on to adapt PETR, enabling better performance with less computational overhead.
  arXiv  Detail & Related papers  (2024-03-08T11:41:48Z)
• PSLT: A Light-weight Vision Transformer with Ladder Self-Attention and Progressive Shift [139.17852337764586]
  Vision Transformer (ViT) has shown great potential for various visual tasks due to its ability to model long-range dependency. We propose a ladder self-attention block with multiple branches and a progressive shift mechanism to develop a light-weight transformer backbone.
  arXiv  Detail & Related papers  (2023-04-07T05:21:37Z)
• Learning to Generate Content-Aware Dynamic Detectors [62.74209921174237]
  We introduce a newpective of designing efficient detectors, which is automatically generating sample-adaptive model architecture. We introduce a course-to-fine strat-egy tailored for object detection to guide the learning of dynamic routing. Experiments on MS-COCO dataset demonstrate that CADDet achieves 1.8 higher mAP with 10% fewer FLOPs compared with vanilla routing.
  arXiv  Detail & Related papers  (2020-12-08T08:05:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.