S-SONDO: Self-Supervised Knowledge Distillation for General Audio Foundation Models

• URL: http://arxiv.org/abs/2604.24933v1
• Date: Mon, 27 Apr 2026 19:20:47 GMT
• Title: S-SONDO: Self-Supervised Knowledge Distillation for General Audio Foundation Models
• Abstract summary: S-SONDO is the first framework to distill general audio models using only their output embeddings.<n>We demonstrate its effectiveness by distilling two audio foundation models into three efficient students.
• Score: 24.103531000455003
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: General audio foundation models have recently achieved remarkable progress, enabling strong performance across diverse tasks. However, state-of-the-art models remain extremely large, often with hundreds of millions of parameters, leading to high inference costs and limited deployability on edge devices. Knowledge distillation is a proven strategy for model compression, but prior work in audio has mostly focused on supervised settings, relying on class logits, intermediate features, or architecture-specific techniques. Such assumptions exclude models that output only embeddings, such as self-supervised or metric-learning models. We introduce S-SONDO (Self-Supervised KnOwledge DistillatioN for General AuDio FOundation Models), the first framework to distill general audio models using only their output embeddings. By avoiding the need for logits or layer-level alignment, S-SONDO is architecture-agnostic and broadly applicable to embedding-based teachers. We demonstrate its effectiveness by distilling two audio foundation models into three efficient students that are up to 61 times smaller while retaining up to 96% of teacher performance. We also provide practical insights on loss choice and clustering-based balanced data sampling. Code is available here: https://github.com/MedAliAdlouni/ssondo.

Related papers

• AVRT: Audio-Visual Reasoning Transfer through Single-Modality Teachers [51.016506778535025]
  We develop a framework that generates high-quality audio-visual reasoning traces from single-modality teacher models.<n>We show that cross-modal training also transfers to single-modality tasks.
  arXiv  Detail & Related papers  (2026-04-17T18:13:27Z)
• Beyond Transcription: Unified Audio Schema for Perception-Aware AudioLLMs [53.41256816230278]
  Recent Audio Large Language Models (AudioLLMs) exhibit a striking performance inversion.<n>While excelling at complex reasoning tasks, they consistently underperform on fine-grained acoustic perception.<n>We propose Unified Audio (UAS), a holistic and structured supervision framework that organizes audio information into three explicit components.
  arXiv  Detail & Related papers  (2026-04-14T09:30:12Z)
• A Human-Inspired Decoupled Architecture for Efficient Audio Representation Learning [0.0]
  We propose HEAR (Human-inspired Efficient Audio Representation), a novel decoupled architecture.<n>HEAR splits the processing pipeline into two dedicated modules: an Acoustic Model for local feature extraction and a Task Model for global semantic integration.<n>Experiments demonstrate that HEAR requires only 15M parameters and 9.47 GFLOPs for inference, operating at a fraction of the computational cost of conventional foundation models.
  arXiv  Detail & Related papers  (2026-03-27T06:09:03Z)
• Foundry: Distilling 3D Foundation Models for the Edge [15.177160184417575]
  Foundation Model Distillation (FMD) is a new paradigm for compressing large SSL models into compact proxies.<n>We present Foundry, the first implementation of FMD for 3D point clouds.
  arXiv  Detail & Related papers  (2025-11-25T07:53:56Z)
• Keep what you need : extracting efficient subnetworks from large audio representation models [0.8798470556253869]
  We introduce learnable binary masks in-between the layers of a pretrained representation model.<n>When training the end-to-end model on a downstream task, we add a sparsity-inducing loss to the overall objective.<n>Once trained, the masked computational units can then be removed from the network, implying significant performance gains.
  arXiv  Detail & Related papers  (2025-02-18T15:04:33Z)
• High-Performance Few-Shot Segmentation with Foundation Models: An Empirical Study [64.06777376676513]
  We develop a few-shot segmentation (FSS) framework based on foundation models. To be specific, we propose a simple approach to extract implicit knowledge from foundation models to construct coarse correspondence. Experiments on two widely used datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2024-09-10T08:04:11Z)
• Leveraging Foundation Models via Knowledge Distillation in Multi-Object Tracking: Distilling DINOv2 Features to FairMOT [0.5999777817331317]
  This work tries to leverage one such foundation model, called DINOv2, through using knowledge distillation. The results imply that although the proposed method shows improvements in certain scenarios, it does not consistently outperform the original FairMOT model.
  arXiv  Detail & Related papers  (2024-07-25T14:21:35Z)
• AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One [47.58919672657824]
  We name this approach AM-RADIO (Agglomerative Model -- Reduce All Domains Into One) We develop a novel architecture (E-RADIO) that exceeds the performance of its predecessors and is at least 7x faster than the teacher models. Our comprehensive benchmarking process covers downstream tasks including ImageNet classification, ADE20k semantic segmentation, COCO object detection and LLaVa-1.5 framework.
  arXiv  Detail & Related papers  (2023-12-10T17:07:29Z)
• Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model [74.62272538148245]
  We show that for arbitrary pairings of pretrained models, one model extracts significant data context unavailable in the other. We investigate if it is possible to transfer such "complementary" knowledge from one model to another without performance degradation.
  arXiv  Detail & Related papers  (2023-10-26T17:59:46Z)
• MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training [74.32603591331718]
  We propose an acoustic Music undERstanding model with large-scale self-supervised Training (MERT), which incorporates teacher models to provide pseudo labels in the masked language modelling (MLM) style acoustic pre-training.<n> Experimental results indicate that our model can generalise and perform well on 14 music understanding tasks and attain state-of-the-art (SOTA) overall scores.
  arXiv  Detail & Related papers  (2023-05-31T18:27:43Z)
• eP-ALM: Efficient Perceptual Augmentation of Language Models [70.47962271121389]
  We propose to direct effort to efficient adaptations of existing models, and propose to augment Language Models with perception. Existing approaches for adapting pretrained models for vision-language tasks still rely on several key components that hinder their efficiency. We show that by freezing more than 99% of total parameters, training only one linear projection layer, and prepending only one trainable token, our approach (dubbed eP-ALM) significantly outperforms other baselines on VQA and Captioning.
  arXiv  Detail & Related papers  (2023-03-20T19:20:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.