AudioScape-TTA: A Structured Soundscape Benchmark for Fine-Grained Text-to-Audio Evaluation

• URL: http://arxiv.org/abs/2608.04479v1
• Date: Wed, 05 Aug 2026 06:06:54 GMT
• Title: AudioScape-TTA: A Structured Soundscape Benchmark for Fine-Grained Text-to-Audio Evaluation
• Abstract summary: We introduce textbfAudioScape-TTA, a structured and complexity-aware benchmark for fine-grained TTA evaluation.<n>The benchmark contains 2,258 audio-text pairs with 25,707 binary QA rubrics, enabling scalable and interpretable analysis.<n>Experiments on 13 representative open-source TTA models reveal persistent limitations in fine-grained attribute control, speech-content preservation, and compositional generation.
• Score: 22.231614260086754
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Text-to-audio (TTA) generation has recently achieved remarkable progress in synthesizing realistic audio from natural language descriptions. However, determining whether generated audio faithfully satisfies complex textual instructions remains challenging. Existing benchmarks mainly rely on global similarity metrics, providing limited insight into fine-grained semantic failures. To address this limitation, we introduce \textbf{AudioScape-TTA}, a structured and complexity-aware benchmark for fine-grained TTA evaluation. AudioScape-TTA represents realistic soundscapes through modality-aware semantic structures and characterizes generation complexity using event density and structural complexity. Based on these annotations, we propose a rubric-based audio-grounded evaluation framework that verifies event realization, acoustic attributes, and speech content through fine-grained semantic criteria. The benchmark contains 2,258 audio-text pairs with 25,707 binary QA rubrics, enabling scalable and interpretable analysis of TTA systems. Experiments on 13 representative open-source TTA models reveal persistent limitations in fine-grained attribute control, speech-content preservation, and compositional soundscape generation. Human validation further demonstrates that our rubric-based evaluation achieves stronger alignment with human semantic judgments than conventional global similarity metrics.

Related papers

• INSPIRE: A Benchmark for Instruction-Aware Speech Retrieval [53.79335341504602]
  INSPIRE is the first benchmark for instruction-aware speech retrieval.<n>We evaluate four retrieval paradigms: large audio-language models, cascaded pipelines, self-supervised speech models, and contrastive audio-language models.
  arXiv  Detail & Related papers  (2026-08-17T07:35:18Z)
• Domain-Specific Evaluation of Text-to-Speech Systems: A Multi-Metric Benchmarking Study [1.224954637705144]
  This paper presents a reproducible, multi-metric benchmarking framework for systematic evaluation of modern TTS systems.<n>Four state-of-the-art TTS systems -- Indic-Parler-TTS, MMS-TTS, Microsoft Edge TTS, and Google Gemini TTS -- are evaluated.<n>Results reveal substantial variation in TTS performance across speech domains, with emotional speech consistently presenting the greatest synthesis challenge.
  arXiv  Detail & Related papers  (2026-08-03T13:49:11Z)
• An Evaluation Framework for Structured Audio Captions Validated by Controlled Perturbations [16.51678229294975]
  We propose a multi-axis evaluation framework tailored for structured audio descriptions.<n>We evaluate outputs across five axes: tag-sets, descriptions, logical reasoning, numeric measurements, and spectral profiles.<n>Our results demonstrate that this framework successfully distinguishes meaning-preserving paraphrases from genuine semantic and acoustic corruptions.
  arXiv  Detail & Related papers  (2026-07-23T15:26:52Z)
• AnyAudio-Judge: A Dynamic Rubric-Based Benchmark and Evaluator for Audio Instruction Following [17.52339026461815]
  We introduce a novel dynamic rubric-based evaluation paradigm that adaptively decomposes complex audio captions into a variable number of independent, verifiable binary rubric items.<n>We benchmark this capability with the AnyAudio-Judge Bench, a comprehensive, bilingual benchmark comprising 7,920 meticulously curated samples.<n>Experiments demonstrate that AnyAudio-Judge not only significantly enhances zero-shot alignment detection compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-06-02T04:00:32Z)
• UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions [55.622295453533475]
  We introduce UniSonate, a unified flow-matching framework capable of synthesizing speech, music, and sound effects.<n>We propose a novel dynamic token injection mechanism that projects unstructured environmental sounds into a structured temporal latent space.<n>Experiments demonstrate that UniSonate achieves state-of-the-art performance in instruction-based TTS and TTM.
  arXiv  Detail & Related papers  (2026-04-24T04:26:04Z)
• Beyond Transcription: Unified Audio Schema for Perception-Aware AudioLLMs [53.41256816230278]
  Recent Audio Large Language Models (AudioLLMs) exhibit a striking performance inversion.<n>While excelling at complex reasoning tasks, they consistently underperform on fine-grained acoustic perception.<n>We propose Unified Audio (UAS), a holistic and structured supervision framework that organizes audio information into three explicit components.
  arXiv  Detail & Related papers  (2026-04-14T09:30:12Z)
• Evaluating Semantic Fragility in Text-to-Audio Generation Systems Under Controlled Prompt Perturbations [2.2870073664564115]
  Small linguistic changes may lead to substantial variation in generated audio, raising concerns about reliability in practical use.<n>We evaluate the semantic fragility of text-to-audio systems under controlled prompt perturbations.
  arXiv  Detail & Related papers  (2026-03-14T08:12:40Z)
• AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answering [97.52852990265136]
  We introduce AQAScore, a backbone-agnostic evaluation framework that leverages the reasoning capabilities of audio-aware large language models.<n>We evaluate AQAScore across multiple benchmarks, including human-rated relevance, pairwise comparison, and compositional reasoning tasks.
  arXiv  Detail & Related papers  (2026-01-21T07:35:36Z)
• WESR: Scaling and Evaluating Word-level Event-Speech Recognition [59.21814194620928]
  Speech conveys not only linguistic information but also rich non-verbal vocal events such as laughing and crying.<n>We develop a refined taxonomy of 21 vocal events, with a new categorization into discrete (standalone) versus continuous (mixed with speech) types.<n>Based on the refined taxonomy, we introduce WESR-Bench, an expert-annotated evaluation set (900+ utterances) with a novel position-aware protocol.
  arXiv  Detail & Related papers  (2026-01-08T02:23:21Z)
• ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing [47.14083940177122]
  ThinkSound is a novel framework that enables stepwise, interactive audio generation and editing for videos.<n>Our approach decomposes the process into three complementary stages: semantically coherent, interactive object-centric refinement, and targeted editing.<n> Experiments demonstrate that ThinkSound achieves state-of-the-art performance in video-to-audio generation across both audio metrics and CoT metrics.
  arXiv  Detail & Related papers  (2025-06-26T16:32:06Z)
• Text-Queried Audio Source Separation via Hierarchical Modeling [53.94434504259829]
  We propose a hierarchical decomposition framework, HSM-TSS, that decouples the task into global-local semantic-guided feature separation and structure-preserving acoustic reconstruction.<n>A Q-Audio architecture is employed to align audio and text modalities, serving as pretrained global-semantic encoders.<n>Our method achieves state-of-the-art separation performance with data-efficient training while maintaining superior semantic consistency with queries in complex auditory scenes.
  arXiv  Detail & Related papers  (2025-05-27T11:00:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.