AnyAudio-Judge: A Dynamic Rubric-Based Benchmark and Evaluator for Audio Instruction Following

• URL: http://arxiv.org/abs/2606.03116v1
• Date: Tue, 02 Jun 2026 04:00:32 GMT
• Title: AnyAudio-Judge: A Dynamic Rubric-Based Benchmark and Evaluator for Audio Instruction Following
• Abstract summary: We introduce a novel dynamic rubric-based evaluation paradigm that adaptively decomposes complex audio captions into a variable number of independent, verifiable binary rubric items.<n>We benchmark this capability with the AnyAudio-Judge Bench, a comprehensive, bilingual benchmark comprising 7,920 meticulously curated samples.<n>Experiments demonstrate that AnyAudio-Judge not only significantly enhances zero-shot alignment detection compared to state-of-the-art baselines.
• Score: 17.52339026461815
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The rapid advancement of instruction-guided audio generation has highlighted the critical need for robust alignment evaluation. Current automated evaluation methods heavily rely on holistic scoring from general-purpose large language models, which struggle to decouple complex instructions, lack interpretability, and fail to capture fine-grained attribute mismatches. To address this, we introduce a novel dynamic rubric-based evaluation paradigm that adaptively decomposes complex audio captions into a variable number of independent, verifiable binary rubric items. To rigorously benchmark this capability, we propose the AnyAudio-Judge Bench, a comprehensive, bilingual benchmark comprising 7,920 meticulously curated samples across four diverse audio domains (speech, sound, music, and mixed), featuring deliberately constructed hard negatives. Furthermore, we construct a large-scale corpus of 105K samples with explicit Chain-of-Thought (CoT) rationales to train our dedicated evaluator, the AnyAudio-Judge model. By employing a training pipeline that combines Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO), our model successfully aligns its reasoning paths with the rubric-based scoring mechanism. Extensive experiments demonstrate that AnyAudio-Judge not only significantly enhances zero-shot alignment detection compared to state-of-the-art baselines, but also provides precise and interpretable reward signals that substantially improve instruction alignment in downstream reinforcement learning for audio generation.

Related papers

• An Evaluation Framework for Structured Audio Captions Validated by Controlled Perturbations [16.51678229294975]
  We propose a multi-axis evaluation framework tailored for structured audio descriptions.<n>We evaluate outputs across five axes: tag-sets, descriptions, logical reasoning, numeric measurements, and spectral profiles.<n>Our results demonstrate that this framework successfully distinguishes meaning-preserving paraphrases from genuine semantic and acoustic corruptions.
  arXiv  Detail & Related papers  (2026-07-23T15:26:52Z)
• Improving Text-to-Audio Instruction Following via Fine-Grained Feedback from Audio-Aware Large Language Models [51.75139461584678]
  Recent text-to-audio models generate high-quality audio, but often fail to follow instructions involving multiple sound events and temporal order.<n>We propose an instruction-level framework that uses audio-aware large language models as fine-grained judges to verify target event presence and temporal relations.<n>Experiments show that our method improves event completeness, temporal ordering, and joint instruction-following accuracy across existing benchmarks and S3Bench, while maintaining audio quality.
  arXiv  Detail & Related papers  (2026-07-15T03:13:06Z)
• JASTIN: Aligning LLMs for Zero-Shot Audio and Speech Evaluation via Natural Language Instructions [66.71387365429062]
  JASTIN is an instruction-driven audio evaluation framework.<n>It formulates audio assessment as a self-instructed reasoning task.<n>It consistently outperforms general MLLMs across speech, sound, music, and out-of-domain evaluation tasks.
  arXiv  Detail & Related papers  (2026-05-06T05:18:42Z)
• Shared Representation Learning for Reference-Guided Targeted Sound Detection [12.339689044414401]
  We propose a unified encoder architecture that processes both the reference and mixture audio within a shared representation space.<n>We establish a new state-of-the-art benchmark for targeted sound detection with a segment-level F1 score of 83.15%.
  arXiv  Detail & Related papers  (2026-03-17T18:05:21Z)
• Representation-Regularized Convolutional Audio Transformer for Audio Understanding [53.092757178419355]
  bootstrapping representations from scratch is computationally expensive, often requiring extensive training to converge.<n>We propose the Convolutional Audio Transformer (CAT), a unified framework designed to address these challenges.
  arXiv  Detail & Related papers  (2026-01-29T12:16:19Z)
• VocSim: A Training-free Benchmark for Zero-shot Content Identity in Single-source Audio [1.0791267046450075]
  VocSim is a training-free benchmark probing the intrinsic geometric alignment of frozen embeddings.<n>VocSim aggregates 125k single-source clips from 19 corpora spanning human speech, animal vocalizations, and environmental sounds.
  arXiv  Detail & Related papers  (2025-12-10T22:13:12Z)
• AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation [55.607230723223346]
  This work presents a systematic study of Large Audio Model (LAM) as a Judge, AudioJudge, investigating whether it can provide a unified evaluation framework that addresses both challenges.<n>We explore AudioJudge across audio characteristic detection tasks, including pronunciation, speaking rate, speaker identification and speech quality, and system-level human preference simulation for automated benchmarking.<n>We introduce a multi-aspect ensemble AudioJudge to enable general-purpose multi-aspect audio evaluation. This method decomposes speech assessment into specialized judges for lexical content, speech quality, and paralinguistic features, achieving up to 0.91 Spearman correlation with human preferences on
  arXiv  Detail & Related papers  (2025-07-17T00:39:18Z)
• $C^2$AV-TSE: Context and Confidence-aware Audio Visual Target Speaker Extraction [80.57232374640911]
  We propose a model-agnostic strategy called the Mask-And-Recover (MAR)<n>MAR integrates both inter- and intra-modality contextual correlations to enable global inference within extraction modules.<n>To better target challenging parts within each sample, we introduce a Fine-grained Confidence Score (FCS) model.
  arXiv  Detail & Related papers  (2025-04-01T13:01:30Z)
• Audio Contrastive-based Fine-tuning: Decoupling Representation Learning and Classification [26.82307246813389]
  We propose a disentangled two-stage framework that separates representation refinement from downstream evaluation.<n>First, we employ a "contrastive-tuning" stage to explicitly improve the geometric structure of the model's embedding space.<n>We then introduce a dual-probe evaluation protocol to assess the quality of these refined representations from a geometric perspective.
  arXiv  Detail & Related papers  (2023-09-21T08:59:13Z)
• Continual Learning for On-Device Speech Recognition using Disentangled Conformers [54.32320258055716]
  We introduce a continual learning benchmark for speaker-specific domain adaptation derived from LibriVox audiobooks. We propose a novel compute-efficient continual learning algorithm called DisentangledCL. Our experiments show that the DisConformer models significantly outperform baselines on general ASR.
  arXiv  Detail & Related papers  (2022-12-02T18:58:51Z)
• Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling [61.351967629600594]
  This paper proposes an any-to-many location-relative, sequence-to-sequence (seq2seq), non-parallel voice conversion approach. In this approach, we combine a bottle-neck feature extractor (BNE) with a seq2seq synthesis module. Objective and subjective evaluations show that the proposed any-to-many approach has superior voice conversion performance in terms of both naturalness and speaker similarity.
  arXiv  Detail & Related papers  (2020-09-06T13:01:06Z)
• Statistical Context-Dependent Units Boundary Correction for Corpus-based Unit-Selection Text-to-Speech [1.4337588659482519]
  We present an innovative technique for speaker adaptation in order to improve the accuracy of segmentation with application to unit-selection Text-To-Speech (TTS) systems. Unlike conventional techniques for speaker adaptation, we aim to use only context dependent characteristics extrapolated with linguistic analysis techniques.
  arXiv  Detail & Related papers  (2020-03-05T12:42:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.