Dynamic Convolutional Neural Networks as Efficient Pre-trained Audio Models

• URL: http://arxiv.org/abs/2310.15648v1
• Date: Tue, 24 Oct 2023 09:08:20 GMT
• Title: Dynamic Convolutional Neural Networks as Efficient Pre-trained Audio Models
• Authors: Florian Schmid, Khaled Koutini, Gerhard Widmer
• Abstract summary: Current popular Audio Spectrogram Transformers are demanding in terms of computational complexity compared to CNNs. We introduce dynamic CNN blocks constructed of dynamic non-linearities, dynamic convolutions and attention mechanisms. Our experiments indicate that the introduced dynamic CNNs achieve better performance on downstream tasks and scale up well.
• Score: 4.803510486360358
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The introduction of large-scale audio datasets, such as AudioSet, paved the way for Transformers to conquer the audio domain and replace CNNs as the state-of-the-art neural network architecture for many tasks. Audio Spectrogram Transformers are excellent at exploiting large datasets, creating powerful pre-trained models that surpass CNNs when fine-tuned on downstream tasks. However, current popular Audio Spectrogram Transformers are demanding in terms of computational complexity compared to CNNs. Recently, we have shown that, by employing Transformer-to-CNN Knowledge Distillation, efficient CNNs can catch up with and even outperform Transformers on large datasets. In this work, we extend this line of research and increase the capacity of efficient CNNs by introducing dynamic CNN blocks, constructed of dynamic non-linearities, dynamic convolutions and attention mechanisms. We show that these dynamic CNNs outperform traditional efficient CNNs, in terms of the performance-complexity trade-off and parameter efficiency, at the task of audio tagging on the large-scale AudioSet. Our experiments further indicate that the introduced dynamic CNNs achieve better performance on downstream tasks and scale up well, attaining Transformer performance and even outperforming them on AudioSet and several downstream tasks.

Related papers

• OA-CNNs: Omni-Adaptive Sparse CNNs for 3D Semantic Segmentation [70.17681136234202]
  We reexamine the design distinctions and test the limits of what a sparse CNN can achieve. We propose two key components, i.e., adaptive receptive fields (spatially) and adaptive relation, to bridge the gap. This exploration led to the creation of Omni-Adaptive 3D CNNs (OA-CNNs), a family of networks that integrates a lightweight module.
  arXiv  Detail & Related papers  (2024-03-21T14:06:38Z)
• The Counterattack of CNNs in Self-Supervised Learning: Larger Kernel Size might be All You Need [103.31261028244782]
  Vision Transformers have been rapidly uprising in computer vision thanks to their outstanding scaling trends, and gradually replacing convolutional neural networks (CNNs) Recent works on self-supervised learning (SSL) introduce siamese pre-training tasks. People come to believe that Transformers or self-attention modules are inherently more suitable than CNNs in the context of SSL.
  arXiv  Detail & Related papers  (2023-12-09T22:23:57Z)
• Transferability of Convolutional Neural Networks in Stationary Learning Tasks [96.00428692404354]
  We introduce a novel framework for efficient training of convolutional neural networks (CNNs) for large-scale spatial problems. We show that a CNN trained on small windows of such signals achieves a nearly performance on much larger windows without retraining. Our results show that the CNN is able to tackle problems with many hundreds of agents after being trained with fewer than ten.
  arXiv  Detail & Related papers  (2023-07-21T13:51:45Z)
• Learning General Audio Representations with Large-Scale Training of Patchout Audio Transformers [6.002503434201551]
  We study the use of audio transformers trained on large-scale datasets to learn general-purpose representations. Our results show that representations extracted by audio transformers outperform CNN representations.
  arXiv  Detail & Related papers  (2022-11-25T08:39:12Z)
• Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation [6.617487928813374]
  We propose a training procedure for efficient CNNs based on offline Knowledge Distillation (KD) from high-performing yet complex transformers. We provide models of different complexity levels, scaling from low-complexity models up to a new state-of-the-art performance of.483 mAP on AudioSet.
  arXiv  Detail & Related papers  (2022-11-09T09:58:22Z)
• Efficient Training of Audio Transformers with Patchout [7.073210405344709]
  We propose a novel method to optimize and regularize transformers on audio spectrograms. The proposed models achieve a new state-of-the-art performance on Audioset and can be trained on a single consumer-grade GPU.
  arXiv  Detail & Related papers  (2021-10-11T08:07:50Z)
• Container: Context Aggregation Network [83.12004501984043]
  Recent finding shows that a simple based solution without any traditional convolutional or Transformer components can produce effective visual representations. We present the model (CONText Ion NERtwok), a general-purpose building block for multi-head context aggregation. In contrast to Transformer-based methods that do not scale well to downstream tasks that rely on larger input image resolutions, our efficient network, named modellight, can be employed in object detection and instance segmentation networks.
  arXiv  Detail & Related papers  (2021-06-02T18:09:11Z)
• Receptive Field Regularization Techniques for Audio Classification and Tagging with Deep Convolutional Neural Networks [7.9495796547433395]
  We show that tuning the Receptive Field (RF) of CNNs is crucial to their generalization. We propose several systematic approaches to control the RF of CNNs and systematically test the resulting architectures.
  arXiv  Detail & Related papers  (2021-05-26T08:36:29Z)
• Exploring Deep Hybrid Tensor-to-Vector Network Architectures for Regression Based Speech Enhancement [53.47564132861866]
  We find that a hybrid architecture, namely CNN-TT, is capable of maintaining a good quality performance with a reduced model parameter size. CNN-TT is composed of several convolutional layers at the bottom for feature extraction to improve speech quality.
  arXiv  Detail & Related papers  (2020-07-25T22:21:05Z)
• Curriculum By Smoothing [52.08553521577014]
  Convolutional Neural Networks (CNNs) have shown impressive performance in computer vision tasks such as image classification, detection, and segmentation. We propose an elegant curriculum based scheme that smoothes the feature embedding of a CNN using anti-aliasing or low-pass filters. As the amount of information in the feature maps increases during training, the network is able to progressively learn better representations of the data.
  arXiv  Detail & Related papers  (2020-03-03T07:27:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.