Face and Voice Cross-modal Association with Learning Convex Feature Embedding

• URL: http://arxiv.org/abs/2607.28129v1
• Date: Thu, 30 Jul 2026 12:37:58 GMT
• Title: Face and Voice Cross-modal Association with Learning Convex Feature Embedding
• Abstract summary: Face-and-voice association learning is one of the most challenging tasks in deep learning.<n>We propose a simple but powerful cross-modal feature embedding method for the association of faces and voices.<n>We exhaustively evaluated our method for cross-modal verification, matching, and retrieval tasks on the large-scale VoxCeleb dataset.
• Score: 10.494561266360858
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Face-and-voice association learning is one of the most challenging tasks in deep learning. In this paper, we propose a simple but powerful cross-modal feature embedding method for the association of faces and voices. Previous work has studied cross-modal association tasks to establish the correlation between voice clips and facial images. These works have addressed cross-modal discrimination but underestimate the importance of handling heterogeneity in inter-modal features between audio and video, resulting in a lot of false positives and false negatives. To tackle the problem, the proposed method learns the embeddings of cross-modal features by making another feature exist between cross-modal features, facilitating the voice and face features of the same person to be embedded in a convex hull. Moreover, the incorporation of cross-modal attention mechanisms with convex embedding techniques represents a highly effective strategy for the attenuation of false positives and false negatives, accomplished via the minimization of inter-class discrepancies. We exhaustively evaluated our method for cross-modal verification, matching, and retrieval tasks on the large-scale VoxCeleb dataset. Extensive experimental results demonstrate that the proposed method achieves notable improvements over existing state-of-the-art methods.

Related papers

• Implicit Counterfactual Learning for Audio-Visual Segmentation [50.69377287012591]
  We propose the implicit counterfactual framework (ICF) to achieve unbiased cross-modal understanding.<n>Due to the lack of semantics, heterogeneous representations may lead to erroneous matches.<n>We introduce the multi-granularity implicit text (MIT) involving video-, segment- and frame-level as the bridge to establish the modality-shared space.
  arXiv  Detail & Related papers  (2025-07-28T11:46:35Z)
• Aligning Sight and Sound: Advanced Sound Source Localization Through Audio-Visual Alignment [50.92136296059296]
  Cross-modal interaction is vital for understanding semantically matched or mismatched audio-visual events. New benchmarks and evaluation metrics reveal previously overlooked issues in sound source localization studies. This work provides the most comprehensive analysis of sound source localization to date.
  arXiv  Detail & Related papers  (2024-07-18T16:51:15Z)
• Learning Unseen Modality Interaction [54.23533023883659]
  Multimodal learning assumes all modality combinations of interest are available during training to learn cross-modal correspondences. We pose the problem of unseen modality interaction and introduce a first solution. It exploits a module that projects the multidimensional features of different modalities into a common space with rich information preserved.
  arXiv  Detail & Related papers  (2023-06-22T10:53:10Z)
• Learning Branched Fusion and Orthogonal Projection for Face-Voice Association [20.973188176888865]
  We propose a light-weight, plug-and-play mechanism that exploits the complementary cues in both modalities to form enriched fused embeddings. Results reveal that our method performs favourably against the current state-of-the-art methods. In addition, we leverage cross-modal verification and matching tasks to analyze the impact of multiple languages on face-voice association.
  arXiv  Detail & Related papers  (2022-08-22T12:23:09Z)
• Probing Visual-Audio Representation for Video Highlight Detection via Hard-Pairs Guided Contrastive Learning [23.472951216815765]
  Key to effective video representations is cross-modal representation learning and fine-grained feature discrimination. In this paper, we enrich intra-modality and cross-modality relations for representation modeling. We enlarge the discriminative power of feature embedding with a hard-pairs guided contrastive learning scheme.
  arXiv  Detail & Related papers  (2022-06-21T07:29:37Z)
• Noise-Tolerant Learning for Audio-Visual Action Recognition [31.641972732424463]
  Video datasets are usually coarse-annotated or collected from the Internet. We propose a noise-tolerant learning framework to find anti-interference model parameters against both noisy labels and noisy correspondence. Our method significantly improves the robustness of the action recognition model and surpasses the baselines by a clear margin.
  arXiv  Detail & Related papers  (2022-05-16T12:14:03Z)
• Unsupervised Voice-Face Representation Learning by Cross-Modal Prototype Contrast [34.58856143210749]
  We present an approach to learn voice-face representations from the talking face videos, without any identity labels. Previous works employ cross-modal instance discrimination tasks to establish the correlation of voice and face. We propose the cross-modal prototype contrastive learning (CMPC), which takes advantage of contrastive methods and resists adverse effects of false negatives and deviate positives.
  arXiv  Detail & Related papers  (2022-04-28T07:28:56Z)
• Dynamic Dual-Attentive Aggregation Learning for Visible-Infrared Person Re-Identification [208.1227090864602]
  Visible-infrared person re-identification (VI-ReID) is a challenging cross-modality pedestrian retrieval problem. Existing VI-ReID methods tend to learn global representations, which have limited discriminability and weak robustness to noisy images. We propose a novel dynamic dual-attentive aggregation (DDAG) learning method by mining both intra-modality part-level and cross-modality graph-level contextual cues for VI-ReID.
  arXiv  Detail & Related papers  (2020-07-18T03:08:13Z)
• Seeing voices and hearing voices: learning discriminative embeddings using cross-modal self-supervision [44.88044155505332]
  We build on earlier work to train embeddings that are more discriminative for uni-modal downstream tasks. We propose a novel training strategy that not only optimises metrics across modalities, but also enforces intra-class feature separation within each of the modalities. The effectiveness of the method is demonstrated on two downstream tasks: lip reading using the features trained on audio-visual synchronisation, and speaker recognition using the features trained for cross-modal biometric matching.
  arXiv  Detail & Related papers  (2020-04-29T16:51:50Z)
• Task-Feature Collaborative Learning with Application to Personalized Attribute Prediction [166.87111665908333]
  We propose a novel multi-task learning method called Task-Feature Collaborative Learning (TFCL) Specifically, we first propose a base model with a heterogeneous block-diagonal structure regularizer to leverage the collaborative grouping of features and tasks. As a practical extension, we extend the base model by allowing overlapping features and differentiating the hard tasks.
  arXiv  Detail & Related papers  (2020-04-29T02:32:04Z)
• Audio-Visual Instance Discrimination with Cross-Modal Agreement [90.95132499006498]
  We present a self-supervised learning approach to learn audio-visual representations from video and audio. We show that optimizing for cross-modal discrimination, rather than within-modal discrimination, is important to learn good representations from video and audio.
  arXiv  Detail & Related papers  (2020-04-27T16:59:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.