Learning Speaker Identity Beyond Language and Modality Constraints: Insights from the POLY-SIM 2026 Challenge

• URL: http://arxiv.org/abs/2607.13669v1
• Date: Wed, 15 Jul 2026 10:12:48 GMT
• Title: Learning Speaker Identity Beyond Language and Modality Constraints: Insights from the POLY-SIM 2026 Challenge
• Abstract summary: Multimodal speaker identification systems assume the availability of complete and homogeneous audio-visual modalities during both training and testing.<n>However, such assumptions often do not hold in real-world applications.
• Score: 26.551250642046764
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal speaker identification systems typically assume the availability of complete and homogeneous audio-visual modalities during both training and testing, and assume each speaker only speaks a single language. However, in real-world applications, such assumptions often do not hold. Visual or audio information may be missing due to occlusions, camera or microphone failures, or privacy constraints. Multilingual speakers introduce additional complexity due to linguistic variability across languages. These situations constitute substantial challenges for the robustness and generalization capabilities of multimodal speaker identification systems. Aim of the POLY-SIM 2026 challenge is to address these aspects of speaker identification and to provide a standardized setup for the comparison of the proposed solutions.

Related papers

• From Speech to Interaction: Analyzing Multimodal Systems in Cocktail-Party Scenarios [68.14493981116665]
  Humans have the remarkable ability to selectively attend to individual speakers while filtering out competing speech from nearby conversations.<n>This "cocktail party" scenario still presents severe challenges to speech recognition systems.<n>The CHiME-9 MCoRec task provides a testbed where systems must recognize groups of speakers and transcribe each of their conversations from audio-visual input.
  arXiv  Detail & Related papers  (2026-08-09T06:11:38Z)
• L-Proto: Language-Aware Episodic Prototypical Training for Multilingual Speaker Verification [49.128847336227636]
  We propose L-Proto, a language-aware episodic training strategy that constructs language-consistent episodes.<n> Experiments on the TidyVoice Challenge benchmark demonstrate consistent performance improvements over conventional fine-tuning and random episodic sampling.
  arXiv  Detail & Related papers  (2026-06-16T01:57:41Z)
• MUSCAT: MUltilingual, SCientific ConversATion Benchmark [60.87925076316812]
  The goal of multilingual speech technology is to facilitate seamless communication between individuals speaking different languages.<n>To create this experience, speech technology needs to address several challenges: Handling mixed multilingual input, specific vocabulary, and code-switching.<n>We propose a new benchmark to evaluate current Automatic Speech Recognition (ASR) systems, whether they are able to handle these challenges.
  arXiv  Detail & Related papers  (2026-04-17T10:39:01Z)
• POLY-SIM: Polyglot Speaker Identification with Missing Modality Grand Challenge 2026 Evaluation Plan [27.26463956543086]
  The Poly-SIM Grand Challenge 2026 aims to advance research in multimodal speaker identification under missing-modality and cross-lingual conditions.<n>Specifically, the Grand Challenge encourages the development of methods that can effectively leverage incomplete multimodal inputs while maintaining strong performance across different languages.
  arXiv  Detail & Related papers  (2026-03-25T17:47:00Z)
• Face-voice Association in Multilingual Environments (FAME) 2026 Challenge Evaluation Plan [24.480174322626155]
  The Face-voice Association in Multilingual Environments (FAME) 2026 Challenge focuses on exploring face-voice association under a multilingual scenario.<n>This report provides the details of the challenge, dataset, baseline models, and task details for the FAME Challenge.
  arXiv  Detail & Related papers  (2025-08-06T16:09:47Z)
• MultiVox: A Benchmark for Evaluating Voice Assistants for Multimodal Interactions [70.93364531054273]
  We introduce MultiVox, the first benchmark to evaluate the ability of voice assistants to integrate spoken and visual cues.<n>Specifically, MultiVox includes 1000 human-annotated and recorded speech dialogues that encompass diverse paralinguistic features.<n>Our evaluation on 10 state-of-the-art models reveals that, although humans excel at these tasks, current models consistently struggle to produce contextually grounded responses.
  arXiv  Detail & Related papers  (2025-07-14T23:20:42Z)
• LASPA: Language Agnostic Speaker Disentanglement with Prefix-Tuned Cross-Attention [2.199918533021483]
  The overlap between vocal traits such as accent, vocal anatomy, and a language's phonetic structure complicates separating linguistic and speaker information.<n>Disentangling these components can significantly improve speaker recognition accuracy.<n>We propose a novel disentanglement learning strategy that integrates joint learning through prefix-tuned cross-attention.
  arXiv  Detail & Related papers  (2025-06-02T10:59:31Z)
• Face-voice Association in Multilingual Environments (FAME) Challenge 2024 Evaluation Plan [29.23176868272216]
  The Face-voice Association in Multilingual Environments (FAME) Challenge 2024 focuses on exploring face-voice association under a unique condition of multilingual scenario. This report provides the details of the challenge, dataset, baselines and task details for the FAME Challenge.
  arXiv  Detail & Related papers  (2024-04-14T19:51:32Z)
• Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyond [87.4049283495551]
  The 2023 Multilingual Speech Universal Performance Benchmark (ML-SUPERB) Challenge expands upon the acclaimed SUPERB framework.<n>The challenge garnered 12 model submissions and 54 language corpora, resulting in a comprehensive benchmark encompassing 154 languages.<n>The findings indicate that merely scaling models is not the definitive solution for multilingual speech tasks.
  arXiv  Detail & Related papers  (2023-10-09T08:30:01Z)
• Investigating on Incorporating Pretrained and Learnable Speaker Representations for Multi-Speaker Multi-Style Text-to-Speech [54.75722224061665]
  In this work, we investigate different speaker representations and proposed to integrate pretrained and learnable speaker representations. The FastSpeech 2 model combined with both pretrained and learnable speaker representations shows great generalization ability on few-shot speakers.
  arXiv  Detail & Related papers  (2021-03-06T10:14:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.