Acoustic Prompting via Stage-wise Modulation for Few-Shot Learning in Audio Language Models

• URL: http://arxiv.org/abs/2606.15751v1
• Date: Sun, 14 Jun 2026 11:23:09 GMT
• Title: Acoustic Prompting via Stage-wise Modulation for Few-Shot Learning in Audio Language Models
• Abstract summary: We propose a framework that introduces trainable prompts into the audio encoder to capture task-specific acoustic features.<n>We demonstrate that integrating audio-side prompt learning with existing text-side approaches enhances few-shot adaptation.
• Score: 57.635707525420884
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Audio-Language Models (ALMs) have shown remarkable success in zero-shot audio classification by aligning audio waveforms with text. Recent efforts to improve downstream performance focus on learning optimal text prompts. However, previous approaches focus on the text encoder, leaving the potential of learnable prompts within the audio encoder unexplored. In this paper, we propose a novel framework that introduces trainable prompts into the audio encoder to capture task-specific acoustic features. We demonstrate that integrating audio-side prompt learning with existing text-side approaches enhances few-shot adaptation. Through extensive experiments across 11 datasets show that integrating our method as a plug-and-play module alongside existing text prompt tuning generally leads to performance improvements. These findings suggest that explicitly modulating the audio representation space effectively complements text-only prompting approaches. The code is available at https://github.com/hyebin-c/aspl.

Related papers

• AudioMosaic: Contrastive Masked Audio Representation Learning [53.52371029884106]
  We introduce textbfAudioMosaic, a contrastive learning-based audio encoder for general audio understanding.<n>AudioMosaic constructs positive pairs by applying structured time-frequency masking to spectrogram patches.<n>Experiments show that AudioMosaic achieves state-of-the-art performance on several standard audio benchmarks.
  arXiv  Detail & Related papers  (2026-05-14T00:56:51Z)
• SeeingSounds: Learning Audio-to-Visual Alignment via Text [15.011814561603964]
  We introduce SeeingSounds, a framework for audio-to-image generation that leverages the interplay between audio, language, and vision.<n>Our method performs dual alignment: audio is projected into a semantic language space via a frozen language encoder, and, contextually grounded into the visual domain using a vision-language model.<n>This approach, inspired by cognitive neuroscience, reflects the natural cross-modal associations observed in human perception.
  arXiv  Detail & Related papers  (2025-10-10T18:42:50Z)
• Enhancing Retrieval-Augmented Audio Captioning with Generation-Assisted Multimodal Querying and Progressive Learning [0.0]
  Generation-Assisted Multimodal Querying generates a text description of the input audio to enable multimodal querying.<n>Our experiments on AudioCaps, Clotho, and Auto-ACD demonstrate that our approach achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2024-10-14T04:57:32Z)
• PALM: Few-Shot Prompt Learning for Audio Language Models [1.6177972328875514]
  Audio-Language Models (ALMs) have recently achieved remarkable success in zero-shot audio recognition tasks. We propose a novel method, Prompt Learning in Audio Language Models (PALM), which optimize the feature space of the text encoder branch. We demonstrate the effectiveness of our approach on 11 audio recognition datasets, and compare the results with three baselines in a few-shot learning setup.
  arXiv  Detail & Related papers  (2024-09-29T22:06:07Z)
• Improving Text-To-Audio Models with Synthetic Captions [51.19111942748637]
  We propose an audio captioning pipeline that uses an textitaudio language model to synthesize accurate and diverse captions for audio at scale. We leverage this pipeline to produce a dataset of synthetic captions for AudioSet, named textttAF-AudioSet, and then evaluate the benefit of pre-training text-to-audio models on these synthetic captions.
  arXiv  Detail & Related papers  (2024-06-18T00:02:15Z)
• Exploring the Role of Audio in Video Captioning [59.679122191706426]
  We present an audio-visual framework, which aims to fully exploit the potential of the audio modality for captioning. We propose new local-global fusion mechanisms to improve information exchange across audio and video.
  arXiv  Detail & Related papers  (2023-06-21T20:54:52Z)
• CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained Language-Vision Models [50.42886595228255]
  We propose to learn the desired text-audio correspondence by leveraging the visual modality as a bridge. We train a conditional diffusion model to generate the audio track of a video, given a video frame encoded by a pretrained contrastive language-image pretraining model.
  arXiv  Detail & Related papers  (2023-06-16T05:42:01Z)
• Unsupervised Audiovisual Synthesis via Exemplar Autoencoders [59.13989658692953]
  We present an unsupervised approach that converts the input speech of any individual into audiovisual streams of potentially-infinitely many output speakers. We use Exemplar Autoencoders to learn the voice, stylistic prosody, and visual appearance of a specific target speech exemplar.
  arXiv  Detail & Related papers  (2020-01-13T18:56:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.