An Evaluation Framework for Structured Audio Captions Validated by Controlled Perturbations

• URL: http://arxiv.org/abs/2607.21424v1
• Date: Thu, 23 Jul 2026 15:26:52 GMT
• Title: An Evaluation Framework for Structured Audio Captions Validated by Controlled Perturbations
• Abstract summary: We propose a multi-axis evaluation framework tailored for structured audio descriptions.<n>We evaluate outputs across five axes: tag-sets, descriptions, logical reasoning, numeric measurements, and spectral profiles.<n>Our results demonstrate that this framework successfully distinguishes meaning-preserving paraphrases from genuine semantic and acoustic corruptions.
• Score: 16.51678229294975
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent advancements in automated audio captioning (AAC) have shifted from monolithic sentence generation toward structured formats that explicitly disentangle distinct acoustic and semantic properties. However, evaluating this heterogeneous data remains a significant challenge. Existing caption metrics focus on flat textual outputs and fail to reliably assess multimodal attributes. To bridge this gap, we propose a multi-axis evaluation framework tailored for structured audio descriptions. Building on the AudioCards dataset, we evaluate outputs across five orthogonal axes: tag-sets, descriptions, logical reasoning, numeric measurements, and spectral profiles. Our approach combines Large Language Model (LLM) judges to capture semantic nuance with deterministic computational metrics to precisely measure acoustic deviations. To rigorously validate the reliability of this framework, we introduce a controlled perturbation testing protocol that injects typed, graded errors into groundtruth annotations. Our results demonstrate that this framework successfully distinguishes meaning-preserving paraphrases from genuine semantic and acoustic corruptions.

Related papers

• Interpreting Object-Dependent Concept Brittleness in Text-to-Image Diffusion Models [105.53190946111586]
  We term this phenomenon object-dependent concept brittleness.<n>We present an interpretability-oriented framework to audit and minimally correct these failures.<n>We evaluate the proposed framework on style and attribute failure cases across multiple diffusion backbones.
  arXiv  Detail & Related papers  (2026-09-09T09:07:40Z)
• AudioScape-TTA: A Structured Soundscape Benchmark for Fine-Grained Text-to-Audio Evaluation [22.231614260086754]
  We introduce textbfAudioScape-TTA, a structured and complexity-aware benchmark for fine-grained TTA evaluation.<n>The benchmark contains 2,258 audio-text pairs with 25,707 binary QA rubrics, enabling scalable and interpretable analysis.<n>Experiments on 13 representative open-source TTA models reveal persistent limitations in fine-grained attribute control, speech-content preservation, and compositional generation.
  arXiv  Detail & Related papers  (2026-08-05T06:06:54Z)
• An Intervention-Based Framework for Shortcut Diagnosis in Spoofing Countermeasures [5.593982292458519]
  We propose an intervention-based diagnostic framework to identify confound-driven shortcut dependencies.<n>We operationalise this through controlled acoustic perturbations targeting non-speech structure, spectral content, and signal energy.<n>Results reveal that non-speech interventions produce the largest performance shifts, confirming non-speech intervals as a dominant shortcut.
  arXiv  Detail & Related papers  (2026-07-03T09:42:31Z)
• Membership Inference Attacks against Large Audio Language Models [50.84901010528239]
  We present the first systematic Membership Inference Attack (MIA) evaluation of Large Audio Language Models (LALMs)<n>Using a multi-modal blind baseline based on textual, spectral, and prosodic features, we demonstrate that common speech datasets exhibit near-perfect train/test separability.<n>These findings establish a principled standard for auditing LALMs beyond spurious correlations.
  arXiv  Detail & Related papers  (2026-03-30T12:45:28Z)
• AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answering [97.52852990265136]
  We introduce AQAScore, a backbone-agnostic evaluation framework that leverages the reasoning capabilities of audio-aware large language models.<n>We evaluate AQAScore across multiple benchmarks, including human-rated relevance, pairwise comparison, and compositional reasoning tasks.
  arXiv  Detail & Related papers  (2026-01-21T07:35:36Z)
• Provable Speech Attributes Conversion via Latent Independence [22.02196595272211]
  We propose a general framework for speech attribute conversion, accompanied by theoretical analysis and guarantees under reasonable assumptions.<n>Our framework builds on a non-probabilistic autoencoder architecture with an independence constraint between the predicted latent variable and the target controllable variable.<n>This design ensures a consistent signal transformation, conditioned on an observed style variable, while preserving the original content and modifying the desired attribute.
  arXiv  Detail & Related papers  (2025-10-06T12:08:27Z)
• $C^2$AV-TSE: Context and Confidence-aware Audio Visual Target Speaker Extraction [80.57232374640911]
  We propose a model-agnostic strategy called the Mask-And-Recover (MAR)<n>MAR integrates both inter- and intra-modality contextual correlations to enable global inference within extraction modules.<n>To better target challenging parts within each sample, we introduce a Fine-grained Confidence Score (FCS) model.
  arXiv  Detail & Related papers  (2025-04-01T13:01:30Z)
• Contextual Biasing to Improve Domain-specific Custom Vocabulary Audio Transcription without Explicit Fine-Tuning of Whisper Model [0.0]
  OpenAI's Whisper Automated Speech Recognition model excels in generalizing across diverse datasets and domains. We propose a method to enhance transcription accuracy without explicit fine-tuning or altering model parameters.
  arXiv  Detail & Related papers  (2024-10-24T01:58:11Z)
• STAB: Speech Tokenizer Assessment Benchmark [57.45234921100835]
  Representing speech as discrete tokens provides a framework for transforming speech into a format that closely resembles text. We present STAB (Speech Tokenizer Assessment Benchmark), a systematic evaluation framework designed to assess speech tokenizers comprehensively. We evaluate the STAB metrics and correlate this with downstream task performance across a range of speech tasks and tokenizer choices.
  arXiv  Detail & Related papers  (2024-09-04T02:20:59Z)
• Learning Disentangled Speech Representations [0.412484724941528]
  SynSpeech is a novel large-scale synthetic speech dataset designed to enable research on disentangled speech representations.<n>We present a framework to evaluate disentangled representation learning techniques, applying both linear probing and established supervised disentanglement metrics.<n>We find that SynSpeech facilitates benchmarking across a range of factors, achieving promising disentanglement of simpler features like gender and speaking style, while highlighting challenges in isolating complex attributes like speaker identity.
  arXiv  Detail & Related papers  (2023-11-04T04:54:17Z)
• Exploiting Sample Uncertainty for Domain Adaptive Person Re-Identification [137.9939571408506]
  We estimate and exploit the credibility of the assigned pseudo-label of each sample to alleviate the influence of noisy labels. Our uncertainty-guided optimization brings significant improvement and achieves the state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2020-12-16T04:09:04Z)
• Continuous speech separation: dataset and analysis [52.10378896407332]
  In natural conversations, a speech signal is continuous, containing both overlapped and overlap-free components. This paper describes a dataset and protocols for evaluating continuous speech separation algorithms.
  arXiv  Detail & Related papers  (2020-01-30T18:01:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.