AVSCap: Orchestrating Audio-Visual Synergy for Omni-modal Video Captioning

• URL: http://arxiv.org/abs/2607.12820v1
• Date: Tue, 14 Jul 2026 14:34:19 GMT
• Title: AVSCap: Orchestrating Audio-Visual Synergy for Omni-modal Video Captioning
• Abstract summary: We present AVSCap, a framework for audio-visual captioning centered on explicit cross-modal event binding.<n>First, we construct AVSCap-130K, a tri-modal training corpus that anchors visual and acoustic evidence before composing grounded omni-modal captions.<n>Second, we train AVSCap-7B, a 7B captioner with a two-stage strategy: supervised fine-tuning establishes baseline capabilities.<n>Third, we introduce AVSCapBench, a benchmark that decomposes captions into visual, audio, and synergy events and evaluates them with fine-grained event recall.
• Score: 45.239264840517585
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Omni-modal video captioning is not merely combining visual captioning with audio transcription: a useful caption must describe how visual actions, speech, music, and sound effects co-evolve. Existing large multimodal models often fail at this relational step, treating audio and visual streams as loosely coupled observations, relying on automatic speech recognition, and under-specifying non-speech sounds and their links to visual events. We present AVSCap, a framework for audio-visual captioning centered on explicit cross-modal event binding. First, we construct AVSCap-130K, a tri-modal training corpus generated by a decoupled-then-fused pipeline that anchors visual and acoustic evidence before composing grounded omni-modal captions. Second, we train AVSCap-7B, a 7B captioner with a two-stage strategy: supervised fine-tuning establishes baseline capabilities, while sample-efficient reinforcement learning uses hybrid rewards to optimize acoustic completeness and audio-visual synergy. Our scaling analysis shows that reinforcement learning brings larger gains than increasing SFT data. Third, we introduce AVSCapBench, a benchmark that decomposes captions into visual, audio, and synergy events and evaluates them with fine-grained event recall. Experiments on AVSCapBench and external benchmarks show that AVSCap-7B improves non-speech audio coverage and cross-modal binding, delivering the best overall performance among evaluated open-source models.

Related papers

• Listening without Looking: Modality Bias in Audio-Visual Captioning [26.155364752676167]
  We conduct modality tests on LAVCap, a state-of-the-art audio-visual captioning model.<n>The analysis reveals a pronounced bias toward the audio stream in LAVCap.<n>We augment AudioCaps with textual annotations that jointly describe the audio and visual streams.<n>The results indicate that LAVCap trained on AudioVisualCaps exhibits less modality bias than when trained on AudioCaps.
  arXiv  Detail & Related papers  (2025-10-28T03:06:28Z)
• LAVCap: LLM-based Audio-Visual Captioning using Optimal Transport [16.108957027494604]
  LAVCap is a large language model (LLM)-based audio-visual captioning framework.<n>It integrates visual information with audio to improve audio captioning performance.<n>It outperforms existing state-of-the-art methods on the AudioCaps dataset.
  arXiv  Detail & Related papers  (2025-01-16T04:53:29Z)
• From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation [17.95017332858846]
  We introduce a novel framework called Vision to Audio and Beyond (VAB) to bridge the gap between audio-visual representation learning and vision-to-audio generation. VAB uses a pre-trained audio tokenizer and an image encoder to obtain audio tokens and visual features, respectively. Our experiments showcase the efficiency of VAB in producing high-quality audio from video, and its capability to acquire semantic audio-visual features.
  arXiv  Detail & Related papers  (2024-09-27T20:26:34Z)
• AVCap: Leveraging Audio-Visual Features as Text Tokens for Captioning [24.608569008975497]
  We propose AVCap, an Audio-Visual Captioning framework. AVCap utilizes audio-visual features as text tokens. Our method outperforms existing audio-visual captioning methods across all metrics.
  arXiv  Detail & Related papers  (2024-07-10T16:17:49Z)
• Bootstrapping Audio-Visual Segmentation by Strengthening Audio Cues [75.73217916395386]
  We propose a Bidirectional Audio-Visual Decoder (BAVD) with integrated bidirectional bridges. This interaction narrows the modality imbalance, facilitating more effective learning of integrated audio-visual representations. We also present a strategy for audio-visual frame-wise synchrony as fine-grained guidance of BAVD.
  arXiv  Detail & Related papers  (2024-02-04T03:02:35Z)
• Exploring the Role of Audio in Video Captioning [59.679122191706426]
  We present an audio-visual framework, which aims to fully exploit the potential of the audio modality for captioning. We propose new local-global fusion mechanisms to improve information exchange across audio and video.
  arXiv  Detail & Related papers  (2023-06-21T20:54:52Z)
• WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research [82.42802570171096]
  We introduce WavCaps, the first large-scale weakly-labelled audio captioning dataset, comprising approximately 400k audio clips with paired captions. Online-harvested raw descriptions are highly noisy and unsuitable for direct use in tasks such as automated audio captioning. We propose a three-stage processing pipeline for filtering noisy data and generating high-quality captions, where ChatGPT, a large language model, is leveraged to filter and transform raw descriptions automatically.
  arXiv  Detail & Related papers  (2023-03-30T14:07:47Z)
• Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention [54.4258176885084]
  How to accurately recognize ambiguous sounds is a major challenge for audio captioning. We propose visually-aware audio captioning, which makes use of visual information to help the description of ambiguous sounding objects. Our proposed method achieves state-of-the-art results on machine translation metrics.
  arXiv  Detail & Related papers  (2022-10-28T22:45:41Z)
• Learning Speech Representations from Raw Audio by Joint Audiovisual Self-Supervision [63.564385139097624]
  We propose a method to learn self-supervised speech representations from the raw audio waveform. We train a raw audio encoder by combining audio-only self-supervision (by predicting informative audio attributes) with visual self-supervision (by generating talking faces from audio) Our results demonstrate the potential of multimodal self-supervision in audiovisual speech for learning good audio representations.
  arXiv  Detail & Related papers  (2020-07-08T14:07:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.