CAF-Score: Calibrating CLAP with LALMs for Reference-free Audio Captioning Evaluation

• URL: http://arxiv.org/abs/2603.19615v1
• Date: Fri, 20 Mar 2026 03:46:18 GMT
• Title: CAF-Score: Calibrating CLAP with LALMs for Reference-free Audio Captioning Evaluation
• Abstract summary: CAF-Score is a reference-free metric that calibrates Contrastive Language-Audio Pretraining.<n>Experiments on the BRACE benchmark demonstrate that our approach achieves the highest correlation with human judgments.
• Score: 8.159381253357127
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While Large Audio-Language Models (LALMs) have advanced audio captioning, robust evaluation remains difficult. Reference-based metrics are expensive and often fail to assess acoustic fidelity, while Contrastive Language-Audio Pretraining (CLAP)-based approaches frequently overlook syntactic errors and fine-grained details. We propose CAF-Score, a reference-free metric that calibrates CLAP's coarse-grained semantic alignment with the fine-grained comprehension and syntactic awareness of LALMs. By combining contrastive audio-text embeddings with LALM reasoning, CAF-Score effectively detects syntactic inconsistencies and subtle hallucinations. Experiments on the BRACE benchmark demonstrate that our approach achieves the highest correlation with human judgments, even outperforming reference-based baselines in challenging scenarios. These results highlight the efficacy of CAF-Score for reference-free audio captioning evaluation. Code and results are available at https://github.com/inseong00/CAF-Score.

Related papers

• Measuring Fairness in Large Audio Language Models via Semantic-Aware Bias Estimation [5.67379336475756]
  Large Audio Language Models (LALMs) have seen increasing use for audio understanding tasks such as speech recognition and audio question answering.<n>We propose a semantic-aware mixed-effects regression framework for fairness evaluation in LALMs.
  arXiv  Detail & Related papers  (2026-08-13T03:24:49Z)
• Membership Inference Attacks against Large Audio Language Models [50.84901010528239]
  We present the first systematic Membership Inference Attack (MIA) evaluation of Large Audio Language Models (LALMs)<n>Using a multi-modal blind baseline based on textual, spectral, and prosodic features, we demonstrate that common speech datasets exhibit near-perfect train/test separability.<n>These findings establish a principled standard for auditing LALMs beyond spurious correlations.
  arXiv  Detail & Related papers  (2026-03-30T12:45:28Z)
• How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation [97.0235251827591]
  Large language models (LLMs) have been widely used as knowledge backbones of Large Audio Language Models (LALMs)<n>We study how much auditory knowledge they encode through text-only pre-training and how this affects downstream performance.<n>Our findings reveal that auditory knowledge varies substantially across families, and text-only results are strongly correlated with audio performance.
  arXiv  Detail & Related papers  (2026-03-19T17:50:07Z)
• AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answering [97.52852990265136]
  We introduce AQAScore, a backbone-agnostic evaluation framework that leverages the reasoning capabilities of audio-aware large language models.<n>We evaluate AQAScore across multiple benchmarks, including human-rated relevance, pairwise comparison, and compositional reasoning tasks.
  arXiv  Detail & Related papers  (2026-01-21T07:35:36Z)
• SEE: Signal Embedding Energy for Quantifying Noise Interference in Large Audio Language Models [49.313324100819955]
  Signal Embedding Energy (SEE) is a method for quantifying the impact of noise intensity on LALM inputs.<n>SEE exhibits a strong correlation with LALM performance, achieving a correlation of 0.98.<n>This paper introduces a novel metric for noise quantification in LALMs, providing guidance for robustness improvements in real-world deployments.
  arXiv  Detail & Related papers  (2026-01-12T08:57:55Z)
• BRACE: A Benchmark for Robust Audio Caption Quality Evaluation [23.704921982469063]
  BRACE is a new benchmark designed to evaluate audio caption alignment quality in a reference-free setting.<n>BRACE consists of two sub-benchmarks: BRACE-Main for fine-grained caption comparison and BRACE-Hallucination for detecting subtle hallucinated content.<n>We evaluate both approaches using the BRACE benchmark, testing CLAPScore across various CLAP model variants and assessing multiple LALMs.
  arXiv  Detail & Related papers  (2025-12-11T08:09:24Z)
• Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samples [55.2480439325792]
  Recent advancements in audio-aware large language models (ALLMs) enable them to process and understand audio inputs.<n>These models often hallucinate non-existent sound events, reducing their reliability in real-world applications.<n>We propose LISTEN, a contrastive-like training method that enhances ALLMs' ability to distinguish between present and absent sounds.
  arXiv  Detail & Related papers  (2025-05-20T15:44:01Z)
• Adapting Pretrained Language Models for Citation Classification via Self-Supervised Contrastive Learning [13.725832389453911]
  Citation classification is pivotal for scholarly analysis.<n>Previous works suggest fine-tuning pretrained language models (PLMs) on citation classification.<n>We present a novel framework, Citss, that adapts the PLMs to overcome these challenges.
  arXiv  Detail & Related papers  (2025-05-20T15:05:27Z)
• CLAIR-A: Leveraging Large Language Models to Judge Audio Captions [73.51087998971418]
  evaluating machine-generated audio captions is a complex task that requires considering diverse factors.<n>We propose CLAIR-A, a simple and flexible method that leverages the zero-shot capabilities of large language models.<n>In our evaluations, CLAIR-A better predicts human judgements of quality compared to traditional metrics.
  arXiv  Detail & Related papers  (2024-09-19T17:59:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.