Input Conditioned Layer Dropping in Speech Foundation Models

• URL: http://arxiv.org/abs/2507.07954v1
• Date: Thu, 10 Jul 2025 17:39:03 GMT
• Title: Input Conditioned Layer Dropping in Speech Foundation Models
• Authors: Abdul Hannan, Daniele Falavigna, Alessio Brutti,
• Abstract summary: layer dropping ($mathcalLD$) skips fraction of the layers of a backbone network during inference to reduce the computational load.<n>We propose input-driven $mathcalLD$ that employs the network's input features and a lightweight layer selecting network to determine optimum combination of processing layers.
• Score: 11.05223262950967
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Curating foundation speech models for edge and IoT settings, where computational resources vary over time, requires dynamic architectures featuring adaptable reduction strategies. One emerging approach is layer dropping ($\mathcal{LD}$) which skips fraction of the layers of a backbone network during inference to reduce the computational load. This allows transforming static models into dynamic ones. However, existing approaches exhibit limitations either in the mode of selecting layers or by significantly modifying the neural architecture. To this end, we propose input-driven $\mathcal{LD}$ that employs the network's input features and a lightweight layer selecting network to determine the optimum combination of processing layers. Extensive experimentation on 4 speech and audio public benchmarks, using two different pre-trained foundation models, demonstrates the effectiveness of our approach, thoroughly outperforming random dropping and producing on-par (or better) results to early exit.

Related papers

• Layer Separation Deep Learning Model with Auxiliary Variables for Partial Differential Equations [0.6215404942415159]
  We propose a new framework, the layer separation (LySep) model, to improve the deep learning-based methods in partial equations.<n>We provide theoretical analyses demonstrating the advantages of LySep in minimizing loss and reducing solution error.
  arXiv  Detail & Related papers  (2025-07-17T03:43:18Z)
• LESA: Learnable LLM Layer Scaling-Up [57.0510934286449]
  Training Large Language Models (LLMs) from scratch requires immense computational resources, making it prohibitively expensive.<n>Model scaling-up offers a promising solution by leveraging the parameters of smaller models to create larger ones.<n>We propose textbfLESA, a novel learnable method for depth scaling-up.
  arXiv  Detail & Related papers  (2025-02-19T14:58:48Z)
• Dynamic layer selection in decoder-only transformers [21.18795712840146]
  We empirically examine two common dynamic inference methods for natural language generation. We find that a pre-trained decoder-only model is significantly more robust to layer removal via layer skipping. We also show that dynamic computation allocation on a per-sequence basis holds promise for significant efficiency gains.
  arXiv  Detail & Related papers  (2024-10-26T00:44:11Z)
• Concurrent Training and Layer Pruning of Deep Neural Networks [0.0]
  We propose an algorithm capable of identifying and eliminating irrelevant layers of a neural network during the early stages of training. We employ a structure using residual connections around nonlinear network sections that allow the flow of information through the network once a nonlinear section is pruned.
  arXiv  Detail & Related papers  (2024-06-06T23:19:57Z)
• Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategy [67.45518210171024]
  Dynamic computation methods have shown notable acceleration for Large Language Models (LLMs) by skipping several layers of computations. We propose a Unified Layer Skipping strategy, which selects the number of layers to skip computation based solely on the target speedup ratio. Experimental results on two common tasks, i.e., machine translation and text summarization, indicate that given a target speedup ratio, the Unified Layer Skipping strategy significantly enhances both the inference performance and the actual model throughput.
  arXiv  Detail & Related papers  (2024-04-10T12:12:07Z)
• Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation [49.44309457870649]
  Layer-wise Feedback feedback (LFP) is a novel training principle for neural network-like predictors.<n>LFP decomposes a reward to individual neurons based on their respective contributions.<n>Our method then implements a greedy reinforcing approach helpful parts of the network and weakening harmful ones.
  arXiv  Detail & Related papers  (2023-08-23T10:48:28Z)
• Layer-wise Linear Mode Connectivity [52.6945036534469]
  Averaging neural network parameters is an intuitive method for the knowledge of two independent models. It is most prominently used in federated learning. We analyse the performance of the models that result from averaging single, or groups.
  arXiv  Detail & Related papers  (2023-07-13T09:39:10Z)
• Efficient Implementation of a Multi-Layer Gradient-Free Online-Trainable Spiking Neural Network on FPGA [0.31498833540989407]
  ODESA is the first network to have end-to-end multi-layer online local supervised training without using gradients. This research shows that the network architecture and the online training of weights and thresholds can be implemented efficiently on a large scale in hardware.
  arXiv  Detail & Related papers  (2023-05-31T00:34:15Z)
• Iterative Soft Shrinkage Learning for Efficient Image Super-Resolution [91.3781512926942]
  Image super-resolution (SR) has witnessed extensive neural network designs from CNN to transformer architectures. This work investigates the potential of network pruning for super-resolution iteration to take advantage of off-the-shelf network designs and reduce the underlying computational overhead. We propose a novel Iterative Soft Shrinkage-Percentage (ISS-P) method by optimizing the sparse structure of a randomly network at each and tweaking unimportant weights with a small amount proportional to the magnitude scale on-the-fly.
  arXiv  Detail & Related papers  (2023-03-16T21:06:13Z)
• Joint inference and input optimization in equilibrium networks [68.63726855991052]
  deep equilibrium model is a class of models that foregoes traditional network depth and instead computes the output of a network by finding the fixed point of a single nonlinear layer. We show that there is a natural synergy between these two settings. We demonstrate this strategy on various tasks such as training generative models while optimizing over latent codes, training models for inverse problems like denoising and inpainting, adversarial training and gradient based meta-learning.
  arXiv  Detail & Related papers  (2021-11-25T19:59:33Z)
• Belief Propagation Reloaded: Learning BP-Layers for Labeling Problems [83.98774574197613]
  We take one of the simplest inference methods, a truncated max-product Belief propagation, and add what is necessary to make it a proper component of a deep learning model. This BP-Layer can be used as the final or an intermediate block in convolutional neural networks (CNNs) The model is applicable to a range of dense prediction problems, is well-trainable and provides parameter-efficient and robust solutions in stereo, optical flow and semantic segmentation.
  arXiv  Detail & Related papers  (2020-03-13T13:11:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.