ASDnB: Merging Face with Body Cues For Robust Active Speaker Detection

• URL: http://arxiv.org/abs/2412.08594v1
• Date: Wed, 11 Dec 2024 18:12:06 GMT
• Title: ASDnB: Merging Face with Body Cues For Robust Active Speaker Detection
• Authors: Tiago Roxo, Joana C. Costa, Pedro Inácio, Hugo Proença,
• Abstract summary: We propose ASDnB, a model that singularly integrates face with body information. Our approach splits 3D convolution into 2D and 1D to reduce computation cost without loss of performance.
• Score: 13.154512864498912
• License:
• Abstract: State-of-the-art Active Speaker Detection (ASD) approaches mainly use audio and facial features as input. However, the main hypothesis in this paper is that body dynamics is also highly correlated to "speaking" (and "listening") actions and should be particularly useful in wild conditions (e.g., surveillance settings), where face cannot be reliably accessed. We propose ASDnB, a model that singularly integrates face with body information by merging the inputs at different steps of feature extraction. Our approach splits 3D convolution into 2D and 1D to reduce computation cost without loss of performance, and is trained with adaptive weight feature importance for improved complement of face with body data. Our experiments show that ASDnB achieves state-of-the-art results in the benchmark dataset (AVA-ActiveSpeaker), in the challenging data of WASD, and in cross-domain settings using Columbia. This way, ASDnB can perform in multiple settings, which is positively regarded as a strong baseline for robust ASD models (code available at https://github.com/Tiago-Roxo/ASDnB).

Related papers

• BIAS: A Body-based Interpretable Active Speaker Approach [13.154512864498912]
  BIAS is a model that combines audio, face, and body information to accurately predict speakers in varying/challenging conditions. Results show that BIAS is state-of-the-art in challenging conditions where body-based features are of utmost importance. BIAS interpretability also shows the features/aspects more relevant towards ASD prediction in varying settings.
  arXiv  Detail & Related papers  (2024-12-06T16:08:09Z)
• ALOcc: Adaptive Lifting-based 3D Semantic Occupancy and Cost Volume-based Flow Prediction [89.89610257714006]
  Existing methods prioritize higher accuracy to cater to the demands of these tasks. We introduce a series of targeted improvements for 3D semantic occupancy prediction and flow estimation. Our purelytemporalal architecture framework, named ALOcc, achieves an optimal tradeoff between speed and accuracy.
  arXiv  Detail & Related papers  (2024-11-12T11:32:56Z)
• Confidence-Aware RGB-D Face Recognition via Virtual Depth Synthesis [48.59382455101753]
  2D face recognition encounters challenges in unconstrained environments due to varying illumination, occlusion, and pose. Recent studies focus on RGB-D face recognition to improve robustness by incorporating depth information. In this work, we first construct a diverse depth dataset generated by 3D Morphable Models for depth model pre-training. Then, we propose a domain-independent pre-training framework that utilizes readily available pre-trained RGB and depth models to separately perform face recognition without needing additional paired data for retraining.
  arXiv  Detail & Related papers  (2024-03-11T09:12:24Z)
• TalkNCE: Improving Active Speaker Detection with Talk-Aware Contrastive Learning [15.673602262069531]
  Active Speaker Detection (ASD) is a task to determine whether a person is speaking or not in a series of video frames. We propose TalkNCE, a novel talk-aware contrastive loss. Our method achieves state-of-the-art performances on AVA-ActiveSpeaker and ASW datasets.
  arXiv  Detail & Related papers  (2023-09-21T17:59:11Z)
• Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memory [64.11870454160614]
  We propose an efficient Adaptive HOI Detector with Concept-guided Memory (ADA-CM) ADA-CM has two operating modes. The first mode makes it tunable without learning new parameters in a training-free paradigm. Our proposed method achieves competitive results with state-of-the-art on the HICO-DET and V-COCO datasets with much less training time.
  arXiv  Detail & Related papers  (2023-09-07T13:10:06Z)
• WASD: A Wilder Active Speaker Detection Dataset [0.0]
  Current Active Speaker Detection (ASD) models achieve great results on AVA-ActiveSpeaker (AVA) using only sound and facial features. We propose a Wilder Active Speaker Detection (WASD) dataset, with increased difficulty by targeting the two key components of current ASD: audio and face. We select state-of-the-art models and assess their performance in two groups of WASD: Easy (cooperative settings) and Hard (audio and/or face are specifically degraded)
  arXiv  Detail & Related papers  (2023-03-09T15:13:22Z)
• A Light Weight Model for Active Speaker Detection [7.253335671577093]
  We construct a lightweight active speaker detection architecture by reducing input candidates, splitting 2D and 3D convolutions for audio-visual feature extraction, and applying gated recurrent unit (GRU) with low computational complexity for cross-modal modeling. Experimental results on the AVA-ActiveSpeaker dataset show that our framework achieves competitive mAP performance (94.1% vs. 94.2%). Our framework also performs well on the Columbia dataset showing good robustness.
  arXiv  Detail & Related papers  (2023-03-08T08:40:56Z)
• Audio-Visual Efficient Conformer for Robust Speech Recognition [91.3755431537592]
  We propose to improve the noise of the recently proposed Efficient Conformer Connectionist Temporal Classification architecture by processing both audio and visual modalities. Our experiments show that using audio and visual modalities allows to better recognize speech in the presence of environmental noise and significantly accelerate training, reaching lower WER with 4 times less training steps.
  arXiv  Detail & Related papers  (2023-01-04T05:36:56Z)
• UniCon: Unified Context Network for Robust Active Speaker Detection [111.90529347692723]
  We introduce a new efficient framework, the Unified Context Network (UniCon), for robust active speaker detection (ASD) Our solution is a novel, unified framework that focuses on jointly modeling multiple types of contextual information. A thorough ablation study is performed on several challenging ASD benchmarks under different settings.
  arXiv  Detail & Related papers  (2021-08-05T13:25:44Z)
• SA-Det3D: Self-Attention Based Context-Aware 3D Object Detection [9.924083358178239]
  We propose two variants of self-attention for contextual modeling in 3D object detection. We first incorporate the pairwise self-attention mechanism into the current state-of-the-art BEV, voxel and point-based detectors. Next, we propose a self-attention variant that samples a subset of the most representative features by learning deformations over randomly sampled locations.
  arXiv  Detail & Related papers  (2021-01-07T18:30:32Z)
• Fine-Grained Vehicle Perception via 3D Part-Guided Visual Data Augmentation [77.60050239225086]
  We propose an effective training data generation process by fitting a 3D car model with dynamic parts to vehicles in real images. Our approach is fully automatic without any human interaction. We present a multi-task network for VUS parsing and a multi-stream network for VHI parsing.
  arXiv  Detail & Related papers  (2020-12-15T03:03:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.