Leveraging Prediction Entropy for Automatic Prompt Weighting in Zero-Shot Audio-Language Classification

• URL: http://arxiv.org/abs/2601.05011v1
• Date: Thu, 08 Jan 2026 15:11:04 GMT
• Title: Leveraging Prediction Entropy for Automatic Prompt Weighting in Zero-Shot Audio-Language Classification
• Abstract summary: We present an entropy-guided prompt weighting approach that aims to find a robust combination of prompt contributions to maximize prediction confidence.<n>Our approach can be applied to individual samples or a batch of audio samples, requiring no additional labels and incurring negligible computational overhead.
• Score: 19.724558192829093
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Audio-language models have recently demonstrated strong zero-shot capabilities by leveraging natural-language supervision to classify audio events without labeled training data. Yet, their performance is highly sensitive to the wording of text prompts, with small variations leading to large fluctuations in accuracy. Prior work has mitigated this issue through prompt learning or prompt ensembling. However, these strategies either require annotated data or fail to account for the fact that some prompts may negatively impact performance. In this work, we present an entropy-guided prompt weighting approach that aims to find a robust combination of prompt contributions to maximize prediction confidence. To this end, we formulate a tailored objective function that minimizes prediction entropy to yield new prompt weights, utilizing low-entropy as a proxy for high confidence. Our approach can be applied to individual samples or a batch of audio samples, requiring no additional labels and incurring negligible computational overhead. Experiments on five audio classification datasets covering environmental, urban, and vocal sounds, demonstrate consistent gains compared to classical prompt ensembling methods in a zero-shot setting, with accuracy improvements 5-times larger across the whole benchmark.

Related papers

• AnyAudio-Judge: A Dynamic Rubric-Based Benchmark and Evaluator for Audio Instruction Following [17.52339026461815]
  We introduce a novel dynamic rubric-based evaluation paradigm that adaptively decomposes complex audio captions into a variable number of independent, verifiable binary rubric items.<n>We benchmark this capability with the AnyAudio-Judge Bench, a comprehensive, bilingual benchmark comprising 7,920 meticulously curated samples.<n>Experiments demonstrate that AnyAudio-Judge not only significantly enhances zero-shot alignment detection compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-06-02T04:00:32Z)
• Adaptive Test-Time Scaling for Zero-Shot Respiratory Audio Classification [14.726932796075927]
  TRIAGE is a tiered zero-shot framework that scales test-time compute by routing each audio sample through richer reasoning stages.<n>A confidence-based router finalizes easy predictions early while allocating additional computation to ambiguous inputs.<n>Across nine respiratory classification tasks without task-specific training, TRIAGE achieves a mean AUROC of 0.744, outperforming prior zero-shot methods.
  arXiv  Detail & Related papers  (2026-04-14T12:17:50Z)
• Generalizable Vision-Language Few-Shot Adaptation with Predictive Prompts and Negative Learning [1.3680468021400563]
  Few-shot adaptation remains a core challenge for vision-language models (VLMs)<n>We propose PromptFuseNL, a unified framework that enhances few-shot generalization by combining predictive prompt tuning with dual-branch positive and negative learning.
  arXiv  Detail & Related papers  (2025-05-16T23:39:34Z)
• Aggregation Strategies for Efficient Annotation of Bioacoustic Sound Events Using Active Learning [0.8678250057211367]
  We introduce Top K Entropy, a novel uncertainty aggregation strategy for Active Learning (AL)<n>Top K Entropy prioritizes the most uncertain segments within an audio recording, instead of averaging uncertainty across all segments.<n>We show that fewer labels can lead to the same model performance, particularly in datasets with sparse sound events.
  arXiv  Detail & Related papers  (2025-03-04T09:08:33Z)
• Vision-Language Models are Strong Noisy Label Detectors [76.07846780815794]
  This paper presents a Denoising Fine-Tuning framework, called DeFT, for adapting vision-language models. DeFT utilizes the robust alignment of textual and visual features pre-trained on millions of auxiliary image-text pairs to sieve out noisy labels. Experimental results on seven synthetic and real-world noisy datasets validate the effectiveness of DeFT in both noisy label detection and image classification.
  arXiv  Detail & Related papers  (2024-09-29T12:55:17Z)
• Foster Adaptivity and Balance in Learning with Noisy Labels [26.309508654960354]
  We propose a novel approach named textbfSED to deal with label noise in a textbfSelf-adaptivtextbfE and class-balancetextbfD manner. A mean-teacher model is then employed to correct labels of noisy samples. We additionally propose a self-adaptive and class-balanced sample re-weighting mechanism to assign different weights to detected noisy samples.
  arXiv  Detail & Related papers  (2024-07-03T03:10:24Z)
• Training-Free Deepfake Voice Recognition by Leveraging Large-Scale Pre-Trained Models [52.04189118767758]
  Generalization is a main issue for current audio deepfake detectors. In this paper we study the potential of large-scale pre-trained models for audio deepfake detection.
  arXiv  Detail & Related papers  (2024-05-03T15:27:11Z)
• Confidence-aware Training of Smoothed Classifiers for Certified Robustness [75.95332266383417]
  We use "accuracy under Gaussian noise" as an easy-to-compute proxy of adversarial robustness for an input. Our experiments show that the proposed method consistently exhibits improved certified robustness upon state-of-the-art training methods.
  arXiv  Detail & Related papers  (2022-12-18T03:57:12Z)
• Checklist Models for Improved Output Fluency in Piano Fingering Prediction [33.52847881359949]
  We present a new approach for the task of predicting fingerings for piano music. We put forward a checklist system, trained via reinforcement learning, that maintains a representation of recent predictions. We demonstrate significant gains in performability directly attributable to improvements with respect to these metrics.
  arXiv  Detail & Related papers  (2022-09-12T21:27:52Z)
• Segment-level Metric Learning for Few-shot Bioacoustic Event Detection [56.59107110017436]
  We propose a segment-level few-shot learning framework that utilizes both the positive and negative events during model optimization. Our system achieves an F-measure of 62.73 on the DCASE 2022 challenge task 5 (DCASE2022-T5) validation set, outperforming the performance of the baseline prototypical network 34.02 by a large margin.
  arXiv  Detail & Related papers  (2022-07-15T22:41:30Z)
• Language Models in the Loop: Incorporating Prompting into Weak Supervision [11.10422546502386]
  We propose a new strategy for applying large pre-trained language models to novel tasks when labeled training data is limited. Instead of applying the model in a typical zero-shot or few-shot fashion, we treat the model as the basis for labeling functions in a weak supervision framework.
  arXiv  Detail & Related papers  (2022-05-04T20:42:40Z)
• Prompt Consistency for Zero-Shot Task Generalization [118.81196556175797]
  In this paper, we explore methods to utilize unlabeled data to improve zero-shot performance. Specifically, we take advantage of the fact that multiple prompts can be used to specify a single task, and propose to regularize prompt consistency. Our approach outperforms the state-of-the-art zero-shot learner, T0, on 9 out of 11 datasets across 4 NLP tasks by up to 10.6 absolute points in terms of accuracy.
  arXiv  Detail & Related papers  (2022-04-29T19:18:37Z)
• Prototypical Classifier for Robust Class-Imbalanced Learning [64.96088324684683]
  We propose textitPrototypical, which does not require fitting additional parameters given the embedding network. Prototypical produces balanced and comparable predictions for all classes even though the training set is class-imbalanced. We test our method on CIFAR-10LT, CIFAR-100LT and Webvision datasets, observing that Prototypical obtains substaintial improvements compared with state of the arts.
  arXiv  Detail & Related papers  (2021-10-22T01:55:01Z)
• End-to-End Adversarial Text-to-Speech [33.01223309795122]
  We learn to synthesise speech from normalised text or phonemes in an end-to-end manner. Our proposed generator is feed-forward and thus efficient for both training and inference. It learns to produce high fidelity audio through a combination of adversarial feedback and prediction losses.
  arXiv  Detail & Related papers  (2020-06-05T17:41:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.