Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsing

• URL: http://arxiv.org/abs/2204.11573v2
• Date: Thu, 28 Apr 2022 04:30:27 GMT
• Title: Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsing
• Authors: Haoyue Cheng, Zhaoyang Liu, Hang Zhou, Chen Qian, Wayne Wu, Limin Wang
• Abstract summary: This paper focuses on the weakly-supervised audio-visual video parsing task. It aims to recognize all events belonging to each modality and localize their temporal boundaries.
• Score: 52.2231419645482
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper focuses on the weakly-supervised audio-visual video parsing task, which aims to recognize all events belonging to each modality and localize their temporal boundaries. This task is challenging because only overall labels indicating the video events are provided for training. However, an event might be labeled but not appear in one of the modalities, which results in a modality-specific noisy label problem. Motivated by two observations that networks tend to learn clean samples first and that a labeled event would appear in at least one modality, we propose a training strategy to identify and remove modality-specific noisy labels dynamically. Specifically, we sort the losses of all instances within a mini-batch individually in each modality, then select noisy samples according to relationships between intra-modal and inter-modal losses. Besides, we also propose a simple but valid noise ratio estimation method by calculating the proportion of instances whose confidence is below a preset threshold. Our method makes large improvements over the previous state of the arts (e.g., from 60.0% to 63.8% in segment-level visual metric), which demonstrates the effectiveness of our approach.

Related papers

• Learning Discriminative Dynamics with Label Corruption for Noisy Label Detection [25.55455239006278]
  We propose DynaCor framework that distinguishes incorrectly labeled instances from correctly labeled ones based on the dynamics of the training signals. Our comprehensive experiments show that DynaCor outperforms the state-of-the-art competitors and shows strong robustness to various noise types and noise rates.
  arXiv  Detail & Related papers  (2024-05-30T10:06:06Z)
• DiffSED: Sound Event Detection with Denoising Diffusion [70.18051526555512]
  We reformulate the SED problem by taking a generative learning perspective. Specifically, we aim to generate sound temporal boundaries from noisy proposals in a denoising diffusion process. During training, our model learns to reverse the noising process by converting noisy latent queries to the groundtruth versions.
  arXiv  Detail & Related papers  (2023-08-14T17:29:41Z)
• Learning with Noisy labels via Self-supervised Adversarial Noisy Masking [33.87292143223425]
  We propose a novel training approach termed adversarial noisy masking. It adaptively modulates the input data and label simultaneously, preventing the model to overfit noisy samples. It is tested on both synthetic and real-world noisy datasets.
  arXiv  Detail & Related papers  (2023-02-14T03:13:26Z)
• Learning from Noisy Labels with Coarse-to-Fine Sample Credibility Modeling [22.62790706276081]
  Training deep neural network (DNN) with noisy labels is practically challenging. Previous efforts tend to handle part or full data in a unified denoising flow. We propose a coarse-to-fine robust learning method called CREMA to handle noisy data in a divide-and-conquer manner.
  arXiv  Detail & Related papers  (2022-08-23T02:06:38Z)
• Noise-Tolerant Learning for Audio-Visual Action Recognition [31.641972732424463]
  Video datasets are usually coarse-annotated or collected from the Internet. We propose a noise-tolerant learning framework to find anti-interference model parameters against both noisy labels and noisy correspondence. Our method significantly improves the robustness of the action recognition model and surpasses the baselines by a clear margin.
  arXiv  Detail & Related papers  (2022-05-16T12:14:03Z)
• S3: Supervised Self-supervised Learning under Label Noise [53.02249460567745]
  In this paper we address the problem of classification in the presence of label noise. In the heart of our method is a sample selection mechanism that relies on the consistency between the annotated label of a sample and the distribution of the labels in its neighborhood in the feature space. Our method significantly surpasses previous methods on both CIFARCIFAR100 with artificial noise and real-world noisy datasets such as WebVision and ANIMAL-10N.
  arXiv  Detail & Related papers  (2021-11-22T15:49:20Z)
• Reliable Shot Identification for Complex Event Detection via Visual-Semantic Embedding [72.9370352430965]
  We propose a visual-semantic guided loss method for event detection in videos. Motivated by curriculum learning, we introduce a negative elastic regularization term to start training the classifier with instances of high reliability. An alternative optimization algorithm is developed to solve the proposed challenging non-net regularization problem.
  arXiv  Detail & Related papers  (2021-10-12T11:46:56Z)
• Unsupervised Sound Localization via Iterative Contrastive Learning [106.56167882750792]
  We propose an iterative contrastive learning framework that requires no data annotations. We then use the pseudo-labels to learn the correlation between the visual and audio signals sampled from the same video. Our iterative strategy gradually encourages the localization of the sounding objects and reduces the correlation between the non-sounding regions and the reference audio.
  arXiv  Detail & Related papers  (2021-04-01T07:48:29Z)
• Tackling Instance-Dependent Label Noise via a Universal Probabilistic Model [80.91927573604438]
  This paper proposes a simple yet universal probabilistic model, which explicitly relates noisy labels to their instances. Experiments on datasets with both synthetic and real-world label noise verify that the proposed method yields significant improvements on robustness.
  arXiv  Detail & Related papers  (2021-01-14T05:43:51Z)
• Unsupervised Contrastive Learning of Sound Event Representations [30.914808451327403]
  Self-supervised representation learning can mitigate the limitations in recognition tasks with few manually labeled data but abundant unlabeled data. In this work, we explore unsupervised contrastive learning as a way to learn sound event representations. Our results suggest that unsupervised contrastive pre-training can mitigate the impact of data scarcity and increase robustness against noisy labels.
  arXiv  Detail & Related papers  (2020-11-15T19:50:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.