EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performance

• URL: http://arxiv.org/abs/2409.01201v1
• Date: Mon, 2 Sep 2024 12:23:18 GMT
• Title: EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performance
• Abstract summary: We analyze and optimize the EnCLAP framework, a state-of-the-art model in automated audio captioning. We develop EnCLAP++, an enhanced version that significantly surpasses the original.
• Score: 4.345953264604522
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we aim to analyze and optimize the EnCLAP framework, a state-of-the-art model in automated audio captioning. We investigate the impact of modifying the acoustic encoder components, explore pretraining with different dataset scales, and study the effectiveness of a reranking scheme. Through extensive experimentation and quantitative analysis of generated captions, we develop EnCLAP++, an enhanced version that significantly surpasses the original.

Related papers

• Impact Analysis of Speech Representation Learning Models for Acoustic Side-Channel Attack [7.2972297703292135]
  KEYAC is a dataset designed to analyze representation generalization for ASCA under both standard and VoIP settings.<n>We evaluate six representation learning models under zero-shot and partial fine-tuning settings using fully connected and convolutional networks.<n>Results show that while partial fine-tuning improves performance, models struggle to generalize across VoIP codecs.
  arXiv  Detail & Related papers  (2026-06-19T08:26:11Z)
• Calibration-Reasoning Framework for Descriptive Speech Quality Assessment [55.51206474234095]
  We introduce a novel post-training method that tailors the foundational Audio Large Language Model for multidimensional reasoning.<n>We reach state-of-the-art results of 0.71 mean PCC score on the multidimensional QualiSpeech benchmark.<n>Our fine-grained GRPO rewards substantially advance the model's ability to pinpoint and classify audio artifacts in time.
  arXiv  Detail & Related papers  (2026-03-10T19:14:17Z)
• Representation-Regularized Convolutional Audio Transformer for Audio Understanding [53.092757178419355]
  bootstrapping representations from scratch is computationally expensive, often requiring extensive training to converge.<n>We propose the Convolutional Audio Transformer (CAT), a unified framework designed to address these challenges.
  arXiv  Detail & Related papers  (2026-01-29T12:16:19Z)
• Rethinking Leveraging Pre-Trained Multi-Layer Representations for Speaker Verification [14.58145497173618]
  We present Layer Attentive Pooling (LAP), a novel strategy for aggregating inter-layer representations from pre-trained speech models for speaker verification.<n>LAP assesses the significance of each layer from multiple perspectives time-dynamically, and employs max pooling instead of averaging.
  arXiv  Detail & Related papers  (2025-12-15T07:39:56Z)
• Improving Model Classification by Optimizing the Training Dataset [3.987352341101438]
  Coresets offer a principled approach to data reduction, enabling efficient learning on large datasets.<n>We present a systematic framework for tuning the coreset generation process to enhance downstream classification quality.
  arXiv  Detail & Related papers  (2025-07-22T16:10:11Z)
• MAO: Efficient Model-Agnostic Optimization of Prompt Tuning for Vision-Language Models [37.85176585188362]
  We propose Model-Agnostic Optimization (MAO) for prompt tuning. We introduce a Data-Driven Enhancement framework to optimize the distribution of the initial data. We incorporate an Alterable Regularization module to boost the task-specific feature processing pipeline.
  arXiv  Detail & Related papers  (2025-03-23T17:59:33Z)
• Semi-supervised Semantic Segmentation with Multi-Constraint Consistency Learning [81.02648336552421]
  We propose a Multi-Constraint Consistency Learning approach to facilitate the staged enhancement of the encoder and decoder. Self-adaptive feature masking and noise injection are designed in an instance-specific manner to perturb the features for robust learning of the decoder. Experimental results on Pascal VOC2012 and Cityscapes datasets demonstrate that our proposed MCCL achieves new state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-03-23T03:21:33Z)
• Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback [50.84142264245052]
  This work introduces the Align-SLM framework to enhance the semantic understanding of textless Spoken Language Models (SLMs) Our approach generates multiple speech continuations from a given prompt and uses semantic metrics to create preference data for Direct Preference Optimization (DPO) We evaluate the framework using ZeroSpeech 2021 benchmarks for lexical and syntactic modeling, the spoken version of the StoryCloze dataset for semantic coherence, and other speech generation metrics, including the GPT4-o score and human evaluation.
  arXiv  Detail & Related papers  (2024-11-04T06:07:53Z)
• Pre-training Feature Guided Diffusion Model for Speech Enhancement [37.88469730135598]
  Speech enhancement significantly improves the clarity and intelligibility of speech in noisy environments. We introduce a novel pretraining feature-guided diffusion model tailored for efficient speech enhancement.
  arXiv  Detail & Related papers  (2024-06-11T18:22:59Z)
• T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretraining [38.604112878493396]
  Contrastive language-audio pretraining(CLAP) has been developed to align the representations of audio and language. We introduce T-CLAP, a temporal-enhanced CLAP model, to capture temporal information within audio and text features. T-CLAP shows improved capability in capturing the temporal relationship of sound events and outperforms state-of-the-art models by a significant margin.
  arXiv  Detail & Related papers  (2024-04-27T07:05:48Z)
• Audio-Visual Generalized Zero-Shot Learning using Pre-Trained Large Multi-Modal Models [53.48409081555687]
  In this work, we explore such large pre-trained models to obtain features, i.e. CLIP for visual features, and CLAP for audio features. We propose a simple yet effective model that only relies on feed-forward neural networks. Our framework achieves state-of-the-art performance on VGGSound-GZSL, UCF-GZSL, and ActivityNet-GZSL.
  arXiv  Detail & Related papers  (2024-04-09T13:39:37Z)
• LAMPER: LanguAge Model and Prompt EngineeRing for zero-shot time series classification [0.4821250031784094]
  This study constructs the LanguAge Model with Prompt EngineeRing (LAMPER) framework to systematically evaluate the adaptability of pre-trained language models (PLMs) in accommodating diverse prompts.
  arXiv  Detail & Related papers  (2024-03-23T15:52:37Z)
• Class Incremental Learning with Pre-trained Vision-Language Models [59.15538370859431]
  We propose an approach to exploiting pre-trained vision-language models (e.g. CLIP) that enables further adaptation. Experiments on several conventional benchmarks consistently show a significant margin of improvement over the current state-of-the-art.
  arXiv  Detail & Related papers  (2023-10-31T10:45:03Z)
• CATfOOD: Counterfactual Augmented Training for Improving Out-of-Domain Performance and Calibration [59.48235003469116]
  We show that data augmentation consistently enhances OOD performance. We also show that CF augmented models which are easier to calibrate also exhibit much lower entropy when assigning importance.
  arXiv  Detail & Related papers  (2023-09-14T16:16:40Z)
• Learning Large-scale Neural Fields via Context Pruned Meta-Learning [60.93679437452872]
  We introduce an efficient optimization-based meta-learning technique for large-scale neural field training. We show how gradient re-scaling at meta-test time allows the learning of extremely high-quality neural fields. Our framework is model-agnostic, intuitive, straightforward to implement, and shows significant reconstruction improvements for a wide range of signals.
  arXiv  Detail & Related papers  (2023-02-01T17:32:16Z)
• A Systematic Comparison of Phonetic Aware Techniques for Speech Enhancement [20.329872147913584]
  We compare different methods of incorporating phonetic information in a speech enhancement model. We observe the influence of different phonetic content models as well as various feature-injection techniques on enhancement performance.
  arXiv  Detail & Related papers  (2022-06-22T12:00:50Z)
• Automatic Audio Captioning using Attention weighted Event based Embeddings [25.258177951665594]
  We propose an encoder-decoder architecture with light-weight (i.e. with lesser learnable parameters) Bi-LSTM recurrent layers for AAC. Our results show that an efficient AED based embedding extractor combined with temporal attention and augmentation techniques is able to surpass existing literature.
  arXiv  Detail & Related papers  (2022-01-28T05:54:19Z)
• Adversarial Feature Augmentation and Normalization for Visual Recognition [109.6834687220478]
  Recent advances in computer vision take advantage of adversarial data augmentation to ameliorate the generalization ability of classification models. Here, we present an effective and efficient alternative that advocates adversarial augmentation on intermediate feature embeddings. We validate the proposed approach across diverse visual recognition tasks with representative backbone networks.
  arXiv  Detail & Related papers  (2021-03-22T20:36:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.