STELLA: Continual Audio-Video Pre-training with Spatio-Temporal Localized Alignment

• URL: http://arxiv.org/abs/2310.08204v3
• Date: Tue, 28 May 2024 15:36:00 GMT
• Title: STELLA: Continual Audio-Video Pre-training with Spatio-Temporal Localized Alignment
• Authors: Jaewoo Lee, Jaehong Yoon, Wonjae Kim, Yunji Kim, Sung Ju Hwang,
• Abstract summary: Continuously learning a variety of audio-video semantics over time is crucial for audio-related reasoning tasks. This is a nontemporal problem and poses two critical challenges: sparse-temporal correlation between audio-video pairs and multimodal correlation overwriting that forgets audio-video relations. We propose a continual audio-video pre-training method with two novel ideas.
• Score: 61.83340833859382
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continuously learning a variety of audio-video semantics over time is crucial for audio-related reasoning tasks in our ever-evolving world. However, this is a nontrivial problem and poses two critical challenges: sparse spatio-temporal correlation between audio-video pairs and multimodal correlation overwriting that forgets audio-video relations. To tackle this problem, we propose a new continual audio-video pre-training method with two novel ideas: (1) Localized Patch Importance Scoring: we introduce a multimodal encoder to determine the importance score for each patch, emphasizing semantically intertwined audio-video patches. (2) Replay-guided Correlation Assessment: to reduce the corruption of previously learned audiovisual knowledge due to drift, we propose to assess the correlation of the current patches on the past steps to identify the patches exhibiting high correlations with the past steps. Based on the results from the two ideas, we perform probabilistic patch selection for effective continual audio-video pre-training. Experimental validation on multiple benchmarks shows that our method achieves a 3.69%p of relative performance gain in zero-shot retrieval tasks compared to strong continual learning baselines, while reducing memory consumption by ~45%.

Related papers

• Missingness-resilient Video-enhanced Multimodal Disfluency Detection [3.3281516035025285]
  We present a practical multimodal disfluency detection approach that leverages available video data together with audio. Our resilient design accommodates real-world scenarios where the video modality may sometimes be missing during inference. In experiments across five disfluency-detection tasks, our unified multimodal approach significantly outperforms Audio-only unimodal methods.
  arXiv  Detail & Related papers  (2024-06-11T05:47:16Z)
• Fuse after Align: Improving Face-Voice Association Learning via Multimodal Encoder [22.836016610542387]
  This paper introduces a novel framework within an unsupervised setting for learning voice-face associations. By employing a multimodal encoder after contrastive learning and addressing the problem through binary classification, we can learn the implicit information within the embeddings in a more effective and varied manner. Empirical evidence demonstrates that our framework achieves state-of-the-art results in voice-face matching, verification, and retrieval tasks.
  arXiv  Detail & Related papers  (2024-04-15T07:05:14Z)
• Speed Co-Augmentation for Unsupervised Audio-Visual Pre-training [102.18680666349806]
  We propose a speed co-augmentation method that randomly changes the playback speeds of both audio and video data. Experimental results show that the proposed method significantly improves the learned representations when compared to vanilla audio-visual contrastive learning.
  arXiv  Detail & Related papers  (2023-09-25T08:22:30Z)
• AdVerb: Visually Guided Audio Dereverberation [49.958724234969445]
  We present AdVerb, a novel audio-visual dereverberation framework. It uses visual cues in addition to the reverberant sound to estimate clean audio.
  arXiv  Detail & Related papers  (2023-08-23T18:20:59Z)
• Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder [58.523884148942166]
  We propose two novel techniques to improve audio-visual speech recognition (AVSR) under a pre-training and fine-tuning training framework. First, we explore the correlation between lip shapes and syllable-level subword units in Mandarin to establish good frame-level syllable boundaries from lip shapes. Next, we propose an audio-guided cross-modal fusion encoder (CMFE) neural network to utilize main training parameters for multiple cross-modal attention layers.
  arXiv  Detail & Related papers  (2023-08-14T08:19:24Z)
• Noise-Tolerant Learning for Audio-Visual Action Recognition [31.641972732424463]
  Video datasets are usually coarse-annotated or collected from the Internet. We propose a noise-tolerant learning framework to find anti-interference model parameters against both noisy labels and noisy correspondence. Our method significantly improves the robustness of the action recognition model and surpasses the baselines by a clear margin.
  arXiv  Detail & Related papers  (2022-05-16T12:14:03Z)
• Recent Advances and Challenges in Deep Audio-Visual Correlation Learning [7.273353828127817]
  This paper focuses on state-of-the-art (SOTA) models used to learn correlations between audio and video. We also discuss some tasks of definition and paradigm applied in AI multimedia.
  arXiv  Detail & Related papers  (2022-02-28T10:43:01Z)
• Audio-Visual Synchronisation in the wild [149.84890978170174]
  We identify and curate a test set with high audio-visual correlation, namely VGG-Sound Sync. We compare a number of transformer-based architectural variants specifically designed to model audio and visual signals of arbitrary length. We set the first benchmark for general audio-visual synchronisation with over 160 diverse classes in the new VGG-Sound Sync video dataset.
  arXiv  Detail & Related papers  (2021-12-08T17:50:26Z)
• Audiovisual transfer learning for audio tagging and sound event detection [21.574781022415372]
  We study the merit of transfer learning for two sound recognition problems, i.e., audio tagging and sound event detection. We adapt a baseline system utilizing only spectral acoustic inputs to make use of pretrained auditory and visual features. We perform experiments with these modified models on an audiovisual multi-label data set.
  arXiv  Detail & Related papers  (2021-06-09T21:55:05Z)
• Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separation [73.1652905564163]
  We address the problem of separating individual speech signals from videos using audio-visual neural processing. Most conventional approaches utilize frame-wise matching criteria to extract shared information between co-occurring audio and video. We propose a cross-modal affinity network (CaffNet) that learns global correspondence as well as locally-varying affinities between audio and visual streams.
  arXiv  Detail & Related papers  (2021-03-25T15:39:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.