Embedding-Space Diffusion for Zero-Shot Environmental Sound Classification

• URL: http://arxiv.org/abs/2412.03771v2
• Date: Tue, 01 Jul 2025 23:59:37 GMT
• Title: Embedding-Space Diffusion for Zero-Shot Environmental Sound Classification
• Abstract summary: Zero-shot learning enables models to generalise to unseen classes by leveraging semantic information.<n>Generative methods, which have demonstrated success in computer vision, are notably absent from zero-shot environmental sound classification studies.<n>This work investigates generative methods for zero-shot learning in environmental audio.
• Score: 44.99833362998488
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Zero-shot learning enables models to generalise to unseen classes by leveraging semantic information, bridging the gap between training and testing sets with non-overlapping classes. While much research has focused on zero-shot learning in computer vision, the application of these methods to environmental audio remains underexplored, with poor performance in existing studies. Generative methods, which have demonstrated success in computer vision, are notably absent from zero-shot environmental sound classification studies. To address this gap, this work investigates generative methods for zero-shot learning in environmental audio. Two successful generative models from computer vision are adapted: a cross-aligned and distribution-aligned variational autoencoder (CADA-VAE) and a leveraging invariant side generative adversarial network (LisGAN). Additionally, we introduced a novel diffusion model conditioned on class auxiliary data. Synthetic embeddings generated by the diffusion model are combined with seen class embeddings to train a classifier. Experiments are conducted on five environmental audio datasets, ESC-50, ARCA23K-FSD, FSC22, UrbanSound8k and TAU Urban Acoustics 2019, and one music classification dataset, GTZAN. Results show that the diffusion model outperforms all baseline methods on average across six audio datasets. This work establishes the diffusion model as a promising approach for zero-shot learning and introduces the first benchmark of generative methods for zero-shot environmental sound classification, providing a foundation for future research.

Related papers

• Acoustically Grounded Cost Learning for Open-Vocabulary Audio-Visual Semantic Segmentation [95.88527433039519]
  Open-Vocabulary Audio-Visual Semantic (OV-AVSS) aims to perform pixel-level segmentation of sound-emitting objects from an open set of categories.<n>We reformulate the objective to be category-specific and propose a novel Acoustically Grounded Cost Learning framework.<n>Experiments on the AVSBench-OV dataset demonstrate that our method significantly outperforms previous state-of-the-art approaches.
  arXiv  Detail & Related papers  (2026-08-29T08:02:27Z)
• Acoustic scattering AI for non-invasive object classifications: A case study on hair assessment [2.5074524384639214]
  This paper presents a novel non-invasive object classification approach using acoustic scattering.<n>We classify hair type and moisture using AI-driven, deep-learning-based sound classification.<n>Results highlight acoustic scattering as a privacy-preserving, non-contact alternative to visual classification.
  arXiv  Detail & Related papers  (2025-06-17T03:25:38Z)
• Extremely Simple Out-of-distribution Detection for Audio-visual Generalized Zero-shot Learning [84.02184773383732]
  Zero-shot Learning attains knowledge transfer from seen classes to unseen classes by exploring auxiliary category information.<n>In this paper, we propose an extremely simple Out-of-distribution(OOD) detection based AV-GZSL method(EZ-AVOOD) to mitigate bias problem.<n>Compared to existing state-of-the-art methods, our model superior ZSL and GZSL performances on three audio-visual datasets.
  arXiv  Detail & Related papers  (2025-03-28T07:28:56Z)
• Joint Diffusion models in Continual Learning [4.013156524547073]
  We introduce JDCL - a new method for continual learning with generative rehearsal based on joint diffusion models. Generative-replay-based continual learning methods try to mitigate this issue by retraining a model with a combination of new and rehearsal data sampled from a generative model. We show that such shared parametrization, combined with the knowledge distillation technique allows for stable adaptation to new tasks without catastrophic forgetting.
  arXiv  Detail & Related papers  (2024-11-12T22:35:44Z)
• ECHO: Environmental Sound Classification with Hierarchical Ontology-guided Semi-Supervised Learning [1.692231043694918]
  We propose a novel framework called Environmental Sound Classification with Hierarchical Ontology-guided semi-supervised Learning (ECHO) In the pretext task, the model tries to predict coarse labels defined by the Large Language Model (LLM) based on ground truth label ontology. Our proposed novel semi-supervised framework achieves an accuracy improvement in the range of 1% to 8% over baseline systems.
  arXiv  Detail & Related papers  (2024-09-21T07:08:57Z)
• Model Inversion Attacks Through Target-Specific Conditional Diffusion Models [54.69008212790426]
  Model inversion attacks (MIAs) aim to reconstruct private images from a target classifier's training set, thereby raising privacy concerns in AI applications. Previous GAN-based MIAs tend to suffer from inferior generative fidelity due to GAN's inherent flaws and biased optimization within latent space. We propose Diffusion-based Model Inversion (Diff-MI) attacks to alleviate these issues.
  arXiv  Detail & Related papers  (2024-07-16T06:38:49Z)
• Diffusion-Model-Assisted Supervised Learning of Generative Models for Density Estimation [10.793646707711442]
  We present a framework for training generative models for density estimation. We use the score-based diffusion model to generate labeled data. Once the labeled data are generated, we can train a simple fully connected neural network to learn the generative model in the supervised manner.
  arXiv  Detail & Related papers  (2023-10-22T23:56:19Z)
• Learning to Reach Goals via Diffusion [16.344212996721346]
  We present a novel perspective on goal-conditioned reinforcement learning by framing it within the context of denoising diffusion models. We then learn a goal-conditioned policy to reverse these deviations, analogous to the score function. This approach, which we call Merlin, can reach specified goals from arbitrary initial states without learning a separate value function.
  arXiv  Detail & Related papers  (2023-10-04T00:47:02Z)
• Bridging the Projection Gap: Overcoming Projection Bias Through Parameterized Distance Learning [9.26015904497319]
  Generalized zero-shot learning (GZSL) aims to recognize samples from both seen and unseen classes using only seen class samples for training. GZSL methods are prone to bias towards seen classes during inference due to the projection function being learned from seen classes. We address this projection bias by proposing to learn a parameterized Mahalanobis distance metric for robust inference.
  arXiv  Detail & Related papers  (2023-09-04T06:41:29Z)
• Continual Zero-Shot Learning through Semantically Guided Generative Random Walks [56.65465792750822]
  We address the challenge of continual zero-shot learning where unseen information is not provided during training, by leveraging generative modeling. We propose our learning algorithm that employs a novel semantically guided Generative Random Walk (GRW) loss. Our algorithm achieves state-of-the-art performance on AWA1, AWA2, CUB, and SUN datasets, surpassing existing CZSL methods by 3-7%.
  arXiv  Detail & Related papers  (2023-08-23T18:10:12Z)
• DiffSED: Sound Event Detection with Denoising Diffusion [70.18051526555512]
  We reformulate the SED problem by taking a generative learning perspective. Specifically, we aim to generate sound temporal boundaries from noisy proposals in a denoising diffusion process. During training, our model learns to reverse the noising process by converting noisy latent queries to the groundtruth versions.
  arXiv  Detail & Related papers  (2023-08-14T17:29:41Z)
• Self-Supervised Visual Acoustic Matching [63.492168778869726]
  Acoustic matching aims to re-synthesize an audio clip to sound as if it were recorded in a target acoustic environment. We propose a self-supervised approach to visual acoustic matching where training samples include only the target scene image and audio. Our approach jointly learns to disentangle room acoustics and re-synthesize audio into the target environment, via a conditional GAN framework and a novel metric.
  arXiv  Detail & Related papers  (2023-07-27T17:59:59Z)
• RanPAC: Random Projections and Pre-trained Models for Continual Learning [59.07316955610658]
  Continual learning (CL) aims to learn different tasks (such as classification) in a non-stationary data stream without forgetting old ones. We propose a concise and effective approach for CL with pre-trained models.
  arXiv  Detail & Related papers  (2023-07-05T12:49:02Z)
• Denoising Diffusion Autoencoders are Unified Self-supervised Learners [58.194184241363175]
  This paper shows that the networks in diffusion models, namely denoising diffusion autoencoders (DDAE), are unified self-supervised learners. DDAE has already learned strongly linear-separable representations within its intermediate layers without auxiliary encoders. Our diffusion-based approach achieves 95.9% and 50.0% linear evaluation accuracies on CIFAR-10 and Tiny-ImageNet.
  arXiv  Detail & Related papers  (2023-03-17T04:20:47Z)
• Learning with Noisy Labels through Learnable Weighting and Centroid Similarity [5.187216033152917]
  noisy labels are prevalent in domains such as medical diagnosis and autonomous driving. We introduce a novel method for training machine learning models in the presence of noisy labels. Our results show that our method consistently outperforms the existing state-of-the-art techniques.
  arXiv  Detail & Related papers  (2023-03-16T16:43:24Z)
• Zero-Shot Logit Adjustment [89.68803484284408]
  Generalized Zero-Shot Learning (GZSL) is a semantic-descriptor-based learning technique. In this paper, we propose a new generation-based technique to enhance the generator's effect while neglecting the improvement of the classifier. Our experiments demonstrate that the proposed technique achieves state-of-the-art when combined with the basic generator, and it can improve various generative zero-shot learning frameworks.
  arXiv  Detail & Related papers  (2022-04-25T17:54:55Z)
• NGC: A Unified Framework for Learning with Open-World Noisy Data [36.96188289965334]
  We propose a new graph-based framework, namely Noisy Graph Cleaning (NGC), which collects clean samples by leveraging geometric structure of data and model predictive confidence. We conduct experiments on multiple benchmarks with different types of noise and the results demonstrate the superior performance of our method against state of the arts.
  arXiv  Detail & Related papers  (2021-08-25T04:04:46Z)
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data [78.69828864672978]
  A central challenge in training classification models in the real-world federated system is learning with non-IID data. We propose a novel and simple algorithm called Virtual Representations (CCVR), which adjusts the classifier using virtual representations sampled from an approximated ssian mixture model. Experimental results demonstrate that CCVR state-of-the-art performance on popular federated learning benchmarks including CIFAR-10, CIFAR-100, and CINIC-10.
  arXiv  Detail & Related papers  (2021-06-09T12:02:29Z)
• Bridging the Gap Between Clean Data Training and Real-World Inference for Spoken Language Understanding [76.89426311082927]
  Existing models are trained on clean data, which causes a textitgap between clean data training and real-world inference. We propose a method from the perspective of domain adaptation, by which both high- and low-quality samples are embedding into similar vector space. Experiments on the widely-used dataset, Snips, and large scale in-house dataset (10 million training examples) demonstrate that this method not only outperforms the baseline models on real-world (noisy) corpus but also enhances the robustness, that is, it produces high-quality results under a noisy environment.
  arXiv  Detail & Related papers  (2021-04-13T17:54:33Z)
• SoundCLR: Contrastive Learning of Representations For Improved Environmental Sound Classification [0.6767885381740952]
  SoundCLR is a supervised contrastive learning method for effective environment sound classification with state-of-the-art performance. Due to the comparatively small sizes of the available environmental sound datasets, we propose and exploit a transfer learning and strong data augmentation pipeline. Our experiments show that our masking based augmentation technique on the log-mel spectrograms can significantly improve the recognition performance.
  arXiv  Detail & Related papers  (2021-03-02T18:42:45Z)
• Imbalanced Data Learning by Minority Class Augmentation using Capsule Adversarial Networks [31.073558420480964]
  We propose a method to restore the balance in imbalanced images, by coalescing two concurrent methods. In our model, generative and discriminative networks play a novel competitive game. The coalescing of capsule-GAN is effective at recognizing highly overlapping classes with much fewer parameters compared with the convolutional-GAN.
  arXiv  Detail & Related papers  (2020-04-05T12:36:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.