Missingness-resilient Video-enhanced Multimodal Disfluency Detection

• URL: http://arxiv.org/abs/2406.06964v1
• Date: Tue, 11 Jun 2024 05:47:16 GMT
• Title: Missingness-resilient Video-enhanced Multimodal Disfluency Detection
• Authors: Payal Mohapatra, Shamika Likhite, Subrata Biswas, Bashima Islam, Qi Zhu,
• Abstract summary: We present a practical multimodal disfluency detection approach that leverages available video data together with audio. Our resilient design accommodates real-world scenarios where the video modality may sometimes be missing during inference. In experiments across five disfluency-detection tasks, our unified multimodal approach significantly outperforms Audio-only unimodal methods.
• Score: 3.3281516035025285
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Most existing speech disfluency detection techniques only rely upon acoustic data. In this work, we present a practical multimodal disfluency detection approach that leverages available video data together with audio. We curate an audiovisual dataset and propose a novel fusion technique with unified weight-sharing modality-agnostic encoders to learn the temporal and semantic context. Our resilient design accommodates real-world scenarios where the video modality may sometimes be missing during inference. We also present alternative fusion strategies when both modalities are assured to be complete. In experiments across five disfluency-detection tasks, our unified multimodal approach significantly outperforms Audio-only unimodal methods, yielding an average absolute improvement of 10% (i.e., 10 percentage point increase) when both video and audio modalities are always available, and 7% even when video modality is missing in half of the samples.

Related papers

• A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition [53.800937914403654]
  Advanced Audio-Visual Speech Recognition (AVSR) systems have been observed to be sensitive to missing video frames. While applying the dropout technique to the video modality enhances robustness to missing frames, it simultaneously results in a performance loss when dealing with complete data input. We propose a novel Multimodal Distribution Approximation with Knowledge Distillation (MDA-KD) framework to reduce over-reliance on the audio modality.
  arXiv  Detail & Related papers  (2024-03-07T06:06:55Z)
• Exploring Missing Modality in Multimodal Egocentric Datasets [89.76463983679058]
  We introduce a novel concept -Missing Modality Token (MMT)-to maintain performance even when modalities are absent. Our method mitigates the performance loss, reducing it from its original $sim 30%$ drop to only $sim 10%$ when half of the test set is modal-incomplete.
  arXiv  Detail & Related papers  (2024-01-21T11:55:42Z)
• AVTENet: Audio-Visual Transformer-based Ensemble Network Exploiting Multiple Experts for Video Deepfake Detection [53.448283629898214]
  The recent proliferation of hyper-realistic deepfake videos has drawn attention to the threat of audio and visual forgeries. Most previous work on detecting AI-generated fake videos only utilize visual modality or audio modality. We propose an Audio-Visual Transformer-based Ensemble Network (AVTENet) framework that considers both acoustic manipulation and visual manipulation.
  arXiv  Detail & Related papers  (2023-10-19T19:01:26Z)
• STELLA: Continual Audio-Video Pre-training with Spatio-Temporal Localized Alignment [61.83340833859382]
  Continuously learning a variety of audio-video semantics over time is crucial for audio-related reasoning tasks. This is a nontemporal problem and poses two critical challenges: sparse-temporal correlation between audio-video pairs and multimodal correlation overwriting that forgets audio-video relations. We propose a continual audio-video pre-training method with two novel ideas.
  arXiv  Detail & Related papers  (2023-10-12T10:50:21Z)
• Flexible-modal Deception Detection with Audio-Visual Adapter [20.6514221670249]
  We propose a novel framework to fuse temporal features across two modalities efficiently. Experiments conducted on two benchmark datasets demonstrate that the proposed method can achieve superior performance.
  arXiv  Detail & Related papers  (2023-02-11T15:47:20Z)
• Multilingual and Multimodal Abuse Detection [3.4352862428120123]
  This paper attempts abuse detection in conversational audio from a multimodal perspective in a multilingual social media setting. Our proposed method, MADA, explicitly focuses on two modalities other than the audio itself. We test the proposed approach on 10 different languages and observe consistent gains in the range 0.6%-5.2% by leveraging multiple modalities.
  arXiv  Detail & Related papers  (2022-04-03T13:28:58Z)
• Learnable Irrelevant Modality Dropout for Multimodal Action Recognition on Modality-Specific Annotated Videos [10.478479158063982]
  We propose a novel framework to effectively leverage the audio modality in vision-specific annotated videos for action recognition. We build a semantic audio-video label dictionary (SAVLD) that maps each video label to its most K-relevant audio labels. We also present a new two-stream video Transformer for efficiently modeling the visual modalities.
  arXiv  Detail & Related papers  (2022-03-06T17:31:06Z)
• Self-attention fusion for audiovisual emotion recognition with incomplete data [103.70855797025689]
  We consider the problem of multimodal data analysis with a use case of audiovisual emotion recognition. We propose an architecture capable of learning from raw data and describe three variants of it with distinct modality fusion mechanisms.
  arXiv  Detail & Related papers  (2022-01-26T18:04:29Z)
• Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval [36.50847375135979]
  Multi-modal learning from video data has seen increased attention recently as it allows to train semantically meaningful embeddings without human annotation. We present a multi-modal, modality fusion transformer approach that learns to exchange information between multiple modalities, such as video, audio, and text, and integrate them into a joined multi-modal representation.
  arXiv  Detail & Related papers  (2021-12-08T18:14:57Z)
• Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing [48.87278703876147]
  A new problem, named audio-visual video parsing, aims to parse a video into temporal event segments and label them as audible, visible, or both. We propose a novel hybrid attention network to explore unimodal and cross-modal temporal contexts simultaneously. Experimental results show that the challenging audio-visual video parsing can be achieved even with only video-level weak labels.
  arXiv  Detail & Related papers  (2020-07-21T01:53:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.