AVCap: Reinforcing Audio-Video Joint Caption with Detail-Aware Reward

• URL: http://arxiv.org/abs/2608.06930v1
• Date: Fri, 07 Aug 2026 08:03:51 GMT
• Title: AVCap: Reinforcing Audio-Video Joint Caption with Detail-Aware Reward
• Abstract summary: We present AVCap-100K, a high-quality dataset of 100K temporally aligned, detail-rich audio-video captions.<n>We also present AVCap-Score, a specialized benchmark and metric for evaluating atomic-level details in audiovisual captions.
• Score: 22.045270667618933
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Detailed audio-video joint captioning is essential for multimodal video understanding and generation. However, prior works are constrained by three main limitations: (1) the scarcity of high-quality public datasets with fine-grained audio-visual joint captions; (2) reinforcement-learning methods that rely on coarse reward signals; and (3) the lack of a benchmark and metric for evaluating detailed audiovisual captions at the atomic level. To address these challenges, we propose: (1) AVCap-100K, a high-quality dataset of 100K temporally aligned, detail-rich audio-video captions; (2) AVCap, a model optimized via Detail-Aware GRPO (Da-GRPO) that achieves state-of-the-art performance among open-source models and matches or surpasses proprietary models on several evaluations; and (3) AVCap-Bench and AVCap-Score, a specialized benchmark and metric for evaluating atomic-level details in audiovisual captions. Our code, models, and datasets are available at https://huggingface.co/collections/Apryle/avcap.

Related papers

• InstructAV2AV: Instruction-Guided Audio-Video Joint Editing [51.67847766136283]
  We propose InstructAV2AV, the first end-to-end framework for instruction-guided audio-video joint editing.<n>We first develop a scalable data synthesis pipeline and construct InsAVE, the first large-scale audio-video editing dataset.<n>InstructAV2AV outperforms state-of-the-art methods across 11 metrics spanning three aspects on two evaluation sets.
  arXiv  Detail & Related papers  (2026-05-18T14:27:05Z)
• AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech [56.08149157180447]
  We introduce AudioCapBench, a benchmark for evaluating audio captioning capabilities of large multimodal models.<n>We evaluate 13 models across two providers (OpenAI, Google Gemini) using both reference-based metrics (METEOR, BLEU, ROUGE-L) and an LLM-as-Judge framework.
  arXiv  Detail & Related papers  (2026-02-27T03:33:37Z)
• Towards Universal Video MLLMs with Attribute-Structured and Quality-Verified Instructions [74.27249614046309]
  ASID-1M is an open-source collection of one million structured, fine-grained audiovisual instruction annotations.<n>ASID-Verify is a scalable data curation pipeline for annotation.<n>ASID-Captioner is a video understanding model trained via Supervised Fine-Tuning.
  arXiv  Detail & Related papers  (2026-02-13T15:20:54Z)
• AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestration [41.31057204656161]
  We present AVoCaDO, a powerful audiovisual video captioner driven by the temporal orchestration between audio and visual modalities.<n> Experimental results demonstrate that AVoCaDO significantly outperforms existing open-source models across four audiovisual video captioning benchmarks.
  arXiv  Detail & Related papers  (2025-10-12T01:20:22Z)
• UGC-VideoCaptioner: An Omni UGC Video Detail Caption Model and New Benchmarks [13.205921806688147]
  Real-world user-generated videos, especially on platforms like TikTok, often feature rich and intertwined audio visual content.<n>Existing video captioning benchmarks and models remain predominantly visual centric, overlooking the crucial role of audio in conveying scene dynamics, speaker intent, and narrative context.<n>To address these challenges, we introduce-VideoCap, a new benchmark and model framework specifically designed for detailed omnimodal captioning of short form user-generated videos.
  arXiv  Detail & Related papers  (2025-07-15T14:08:29Z)
• Quality Over Quantity? LLM-Based Curation for a Data-Efficient Audio-Video Foundation Model [16.22530358172138]
  The framework implements a scoring mechanism for selecting aligned training data segments.<n>It integrates Whisper, a speech-based foundation model, for audio and DINOv2 for video analysis in a dual-encoder structure.<n> Evaluations on AudioCaps, VALOR, and VGGSound demonstrate the effectiveness of the proposed model architecture.
  arXiv  Detail & Related papers  (2025-03-12T09:48:38Z)
• Taming Data and Transformers for Audio Generation [31.815960560115176]
  AutoReCap-XL is the largest ambient audio-text dataset with over 47 million clips.<n>AutoCap is a high-quality automatic audio captioning model.<n>GenAu is a scalable transformer-based audio generation architecture.
  arXiv  Detail & Related papers  (2024-06-27T17:58:54Z)
• Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learning [50.28566759231076]
  We propose an innovative, automatic approach to establish an audio dataset with high-quality captions. Specifically, we construct a large-scale, high-quality, audio-language dataset, named as Auto-ACD, comprising over 1.5M audio-text pairs. We employ LLM to paraphrase a congruent caption for each audio, guided by the extracted multi-modality clues.
  arXiv  Detail & Related papers  (2023-09-20T17:59:32Z)
• Text-to-feature diffusion for audio-visual few-shot learning [59.45164042078649]
  Few-shot learning from video data is a challenging and underexplored, yet much cheaper, setup. We introduce a unified audio-visual few-shot video classification benchmark on three datasets. We show that AV-DIFF obtains state-of-the-art performance on our proposed benchmark for audio-visual few-shot learning.
  arXiv  Detail & Related papers  (2023-09-07T17:30:36Z)
• AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representations [88.30635799280923]
  We introduce AV-data2vec which builds audio-visual representations based on predicting contextualized representations. Results on LRS3 show that AV-data2vec consistently outperforms existing methods with the same amount of data and model size.
  arXiv  Detail & Related papers  (2023-02-10T02:55:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.