Knowing When Not to Answer: Pseudo-Ensembles for Abstention in Music Audio-Language Models

• URL: http://arxiv.org/abs/2609.04362v1
• Date: Thu, 03 Sep 2026 18:21:25 GMT
• Title: Knowing When Not to Answer: Pseudo-Ensembles for Abstention in Music Audio-Language Models
• Abstract summary: Music audio-language models are evaluated almost entirely by accuracy on multiple-choice questions.<n>What is missing is a way to tell when the model does not know the answer, so that it can abstain instead of guessing.<n>We build pseudo-ensembles from one pretrained model by perturbing its input in ways that cannot change the correct answer.
• Score: 3.6463026920885824
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Music audio-language models are evaluated almost entirely by accuracy on multiple-choice questions. This protocol forces the model to commit to an option, so a lucky guess looks the same as real musical understanding. What is missing is a way to tell when the model does not know the answer, so that it can abstain instead of guessing. The usual solution, an ensemble of independently trained models, is far too expensive here, which leaves the entropy of a single predictive distribution as the only available confidence signal. We instead build pseudo-ensembles from one pretrained model by perturbing its input in ways that cannot change the correct answer, then averaging the resulting distributions over the options. Our main construction simply shuffles the order in which the candidate answers are presented; we also study ensembles built from corrupted audio and from swapped option labels. A pseudo-ensemble gives several predictive distributions per question, so it supports the full family of ensemble-based uncertainty measures (entropy of the expected distribution, expected entropy, and their difference, the mutual information) rather than entropy alone. Evaluating TinyMU on MuChoMusic, we find that averaging over four option orderings raises accuracy from 55.7% to 59.2%, and that the resulting uncertainty measures rank the model's errors better than the single-pass entropy baseline, reducing the area under the error retention curve from 0.293 to 0.261. All of this costs a few extra forward passes and no retraining, which makes abstention practical for compact music audio-language models.

Related papers

• Measuring Reward Hacking and Reasoning-Answer Decoupling Under Position-Confounded Optimization [1.1316247605466565]
  We train language models with GRPO on math problems where the correct answer is always option A, then evaluate on an unseen test set with unbiased answer positions.<n>Across Qwen2.5, Llama 3.x and Gemma-3 models, biased training often drives option-A rates above 0.90 in smaller models.<n>We track reasoning-answer decoupling: capable models generate reasoning that reaches the correct numeric answer while still selecting A.
  arXiv  Detail & Related papers  (2026-08-15T23:13:57Z)
• Sample-Adaptive Latent Rewards for Uncertainty-Guided Diffusion Post-Training [51.18425726646089]
  We propose textscSURE, a unified latent-space framework for image and video diffusion models.<n>It learns reward distributions and directly uses their reliability to guide dense post-training.<n>textscSURE-REFL achieves the highest VBench quality, semantic, and total scores among the evaluated methods.
  arXiv  Detail & Related papers  (2026-08-06T14:55:42Z)
• Instruction-Tuned Language Models Cannot Sample from Distributions They Can Describe [2.6646205080230536]
  Silicon sampling uses language models as proxies for human survey respondents.<n>We show this draw does not exist: instruction-tuned models do not sample from distributions, they collapse to a single output.<n>We propose Prompt-Perturbed Argyle, which reduces the same error by 21% at no added cost.
  arXiv  Detail & Related papers  (2026-07-28T04:58:46Z)
• AB-RAG: Adaptive Budgeted Retrieval-Augmented Generation for Reliable Question Answering [0.0]
  AB-RAG is a training-free and backbone-agnostic framework that generates an answer.<n>It estimates its confidence from a combination of three signals.<n>It then decides whether to stop or to retrieve more evidence.
  arXiv  Detail & Related papers  (2026-06-27T21:08:14Z)
• SimpleStrat: Diversifying Language Model Generation with Stratification [26.933029655072488]
  Prior approaches rely on increasing temperature to increase diversity. We show this approach produces lower quality individual generations as temperature increases. We propose SimpleStrat, an alternative approach that uses the language model itself to partition the space into strata.
  arXiv  Detail & Related papers  (2024-10-11T17:54:14Z)
• DistPred: A Distribution-Free Probabilistic Inference Method for Regression and Forecasting [14.390842560217743]
  We propose a novel approach called DistPred for regression and forecasting tasks.<n>We transform proper scoring rules that measure the discrepancy between the predicted distribution and the target distribution into a differentiable discrete form.<n>This allows the model to sample numerous samples in a single forward pass to estimate the potential distribution of the response variable.
  arXiv  Detail & Related papers  (2024-06-17T10:33:00Z)
• Reconciling Model Multiplicity for Downstream Decision Making [24.335927243672952]
  We show that even when the two predictive models approximately agree on their individual predictions almost everywhere, it is still possible for their induced best-response actions to differ on a substantial portion of the population. We propose a framework that calibrates the predictive models with regard to both the downstream decision-making problem and the individual probability prediction.
  arXiv  Detail & Related papers  (2024-05-30T03:36:46Z)
• Confidence-Based Model Selection: When to Take Shortcuts for Subpopulation Shifts [119.22672589020394]
  We propose COnfidence-baSed MOdel Selection (CosMoS), where model confidence can effectively guide model selection. We evaluate CosMoS on four datasets with spurious correlations, each with multiple test sets with varying levels of data distribution shift.
  arXiv  Detail & Related papers  (2023-06-19T18:48:15Z)
• Composed Image Retrieval with Text Feedback via Multi-grained Uncertainty Regularization [73.04187954213471]
  We introduce a unified learning approach to simultaneously modeling the coarse- and fine-grained retrieval. The proposed method has achieved +4.03%, +3.38%, and +2.40% Recall@50 accuracy over a strong baseline.
  arXiv  Detail & Related papers  (2022-11-14T14:25:40Z)
• How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering [80.82194311274694]
  We examine the question "how can we know when language models know, with confidence, the answer to a particular query?" We examine three strong generative models -- T5, BART, and GPT-2 -- and study whether their probabilities on QA tasks are well calibrated. We then examine methods to calibrate such models to make their confidence scores correlate better with the likelihood of correctness.
  arXiv  Detail & Related papers  (2020-12-02T03:53:13Z)
• Distributional Reinforcement Learning via Moment Matching [54.16108052278444]
  We formulate a method that learns a finite set of statistics from each return distribution via neural networks. Our method can be interpreted as implicitly matching all orders of moments between a return distribution and its Bellman target. Experiments on the suite of Atari games show that our method outperforms the standard distributional RL baselines.
  arXiv  Detail & Related papers  (2020-07-24T05:18:17Z)
• Selective Question Answering under Domain Shift [90.021577320085]
  Abstention policies based solely on the model's softmax probabilities fare poorly, since models are overconfident on out-of-domain inputs. We train a calibrator to identify inputs on which the QA model errs, and abstain when it predicts an error is likely. Our method answers 56% of questions while maintaining 80% accuracy; in contrast, directly using the model's probabilities only answers 48% at 80% accuracy.
  arXiv  Detail & Related papers  (2020-06-16T19:13:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.