Effective Distillation to Hybrid xLSTM Architectures

• URL: http://arxiv.org/abs/2603.15590v1
• Date: Mon, 16 Mar 2026 17:49:04 GMT
• Title: Effective Distillation to Hybrid xLSTM Architectures
• Abstract summary: We introduce an effective distillation pipeline for xLSTM-based students.<n>We show the effectiveness of this pipeline by distilling base and instruction-tuned models from the Llama, Qwen, and Olmo families.
• Score: 24.88882332108881
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: There have been numerous attempts to distill quadratic attention-based large language models (LLMs) into sub-quadratic linearized architectures. However, despite extensive research, such distilled models often fail to match the performance of their teacher LLMs on various downstream tasks. We set out the goal of lossless distillation, which we define in terms of tolerance-corrected Win-and-Tie rates between student and teacher on sets of tasks. To this end, we introduce an effective distillation pipeline for xLSTM-based students. We propose an additional merging stage, where individually linearized experts are combined into a single model. We show the effectiveness of this pipeline by distilling base and instruction-tuned models from the Llama, Qwen, and Olmo families. In many settings, our xLSTM-based students recover most of the teacher's performance, and even exceed it on some downstream tasks. Our contributions are an important step towards more energy-efficient and cost-effective replacements for transformer-based LLMs.

Related papers

• Flexible Feature Distillation for Large Language Models [4.277471273303775]
  We propose a parameter-free framework for task-driven feature distillation for large language models (LLMs)<n>Instead of projecting the entire teacher representation, Flex-KD uses gradient-based scores to identify the most task-relevant dimensions of the teacher's hidden states.<n>Experiments show that Flex-KD consistently boosts student performance, achieving up to a 3.75 percent performance gain over the linear projection baseline.
  arXiv  Detail & Related papers  (2025-07-14T11:10:02Z)
• Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation [64.15918654558816]
  Self-distillation (SSD) training strategy is introduced for filtering and weighting teacher representation to distill from task-relevant representations only.<n> Experimental results on real-world affective computing, wearable/biosignal datasets from the UCR Archive, the HAR dataset, and image classification datasets show that the proposed SSD method can outperform state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-04-19T14:08:56Z)
• DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs [58.4911494598431]
  DistiLLM-2 is a contrastive approach that simultaneously increases the likelihood of teacher responses and decreases that of student responses.<n>Our experiments show that DistiLLM-2 not only builds high-performing student models across a wide range of tasks, but also supports diverse applications.
  arXiv  Detail & Related papers  (2025-03-10T08:51:32Z)
• Pre-training Distillation for Large Language Models: A Design Space Exploration [54.67324039434781]
  Pre-training distillation aims to transfer knowledge from a large teacher model to a smaller student model. We conduct experiments to explore the design space of pre-training distillation and find better configurations. We hope our exploration of the design space will inform future practices in pre-training distillation.
  arXiv  Detail & Related papers  (2024-10-21T17:16:13Z)
• LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillation [41.05687297326706]
  LLaVA-MoD is a framework designed to enable the efficient training of small-scale Multimodal Language Models. We optimize the network structure of s-MLLM by integrating a sparse Mixture of Experts architecture into the language model. We also propose a progressive knowledge transfer strategy to ensure comprehensive knowledge migration.
  arXiv  Detail & Related papers  (2024-08-28T15:52:23Z)
• DDK: Distilling Domain Knowledge for Efficient Large Language Models [40.839056203329136]
  Knowledge Distillation (KD) has emerged as an effective strategy to improve the performance of a smaller language model. This paper introduces DDK, which adjusts the composition of the distillation dataset according to the domain performance differences between the teacher and student models. Extensive evaluations show that DDK significantly improves the performance of student models, outperforming both continuously pretrained baselines and existing knowledge distillation methods by a large margin.
  arXiv  Detail & Related papers  (2024-07-23T03:47:28Z)
• Multi-Granularity Semantic Revision for Large Language Model Distillation [66.03746866578274]
  We propose a multi-granularity semantic revision method for LLM distillation. At the sequence level, we propose a sequence correction and re-generation strategy. At the token level, we design a distribution adaptive clipping Kullback-Leibler loss as the distillation objective function. At the span level, we leverage the span priors of a sequence to compute the probability correlations within spans, and constrain the teacher and student's probability correlations to be consistent.
  arXiv  Detail & Related papers  (2024-07-14T03:51:49Z)
• PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairs [47.35598271306371]
  Large Language Models (LLMs) have exhibited impressive capabilities in various tasks, yet their vast parameter sizes restrict their applicability in resource-constrained settings. Knowledge distillation (KD) offers a viable solution by transferring expertise from large teacher models to compact student models. We present PLaD, a novel preference-based LLM distillation framework.
  arXiv  Detail & Related papers  (2024-06-05T03:08:25Z)
• Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs [17.564185128246272]
  Knowledge distillation offers a solution by compressing knowledge from resource-intensive large models to smaller ones.<n>We introduce Universal Logit Distillation (ULD) loss, grounded in optimal transport, to address this limitation.
  arXiv  Detail & Related papers  (2024-02-19T10:37:29Z)
• Progressive Distillation Based on Masked Generation Feature Method for Knowledge Graph Completion [29.297959023968165]
  This paper proposes a progressive distillation method based on masked generation features for KGC task. Specifically, we perform pre-distillation on PLM to obtain high-quality teacher models, and compress the PLM network to obtain multi-grade student models. The experimental results demonstrate that the model in the pre-distillation stage surpasses the existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-01-19T07:34:36Z)
• Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners [102.20090188997301]
  We explore how to obtain a model that combines Contrastive Learning (CL) and Masked Image Modeling (MIM) strengths. In order to better obtain both discrimination and diversity, we propose a simple but effective Hybrid Distillation strategy. Experiment results prove that Hybrid Distill can achieve superior performance on different benchmarks.
  arXiv  Detail & Related papers  (2023-06-28T02:19:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.