CHALIS: A Challenge Dataset for Language Identification in Difficult Scenarios

• URL: http://arxiv.org/abs/2606.06088v1
• Date: Thu, 04 Jun 2026 12:26:19 GMT
• Title: CHALIS: A Challenge Dataset for Language Identification in Difficult Scenarios
• Abstract summary: CHALIS (Challenging Language Identification Samples) is a new benchmark dataset designed to address difficult cases in language identification.<n>Our dataset has two parts: First, we collected sentences shared across mutually intelligible language pairs (Czech/Slovak, Spanish/Catalan, Portuguese/Galician, Danish/Norwegian)<n>The second part tests for orthography noise: we transliterate text across multiple scripts, remove diacritics, simulate homoglyph attacks, and use Internet slang.
• Score: 0.35604294978773265
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present CHALIS (Challenging Language Identification Samples), a new benchmark dataset explicitly designed to address difficult cases in language identification: cousin languages and orthographic noise. Our dataset has two parts: First, we collected sentences shared across mutually intelligible language pairs (Czech/Slovak, Spanish/Catalan, Portuguese/Galician, Danish/Norwegian). The second part tests for orthography noise: we transliterate text across multiple scripts, remove diacritics, simulate homoglyph attacks, and use Internet slang. We evaluate four widely used language identification systems on CHALIS and demonstrate that all struggle substantially in these scenarios, especially on lower-resource languages within cousin pairs and on transliterated input. The resource is publicly available at https://huggingface.co/datasets/michal-tichy/CHALIS.

Related papers

• MUSCAT: MUltilingual, SCientific ConversATion Benchmark [60.87925076316812]
  The goal of multilingual speech technology is to facilitate seamless communication between individuals speaking different languages.<n>To create this experience, speech technology needs to address several challenges: Handling mixed multilingual input, specific vocabulary, and code-switching.<n>We propose a new benchmark to evaluate current Automatic Speech Recognition (ASR) systems, whether they are able to handle these challenges.
  arXiv  Detail & Related papers  (2026-04-17T10:39:01Z)
• DIVERS-Bench: Evaluating Language Identification Across Domain Shifts and Code-Switching [8.14614722074297]
  Language Identification (LID) is a core task in multilingual NLP, yet current systems often overfit to clean, monolingual data.<n>This work introduces DIVERS-BENCH, a comprehensive evaluation of state-of-the-art LID models across diverse domains.<n>Our findings reveal that while models achieve high accuracy on curated datasets, performance degrades sharply on noisy and informal inputs.
  arXiv  Detail & Related papers  (2025-09-22T13:32:31Z)
• ILID: Native Script Language Identification for Indian Languages [0.0]
  Core challenge of language identification lies in distinguishing languages in noisy, short, and code-mixed environments.<n>We release a dataset of 250K sentences consisting of 23 languages including English and all 22 official Indian languages labeled with their language identifiers.<n>Our models outperforms the state-of-the-art pre-trained transformer models for the language identification task.
  arXiv  Detail & Related papers  (2025-07-16T01:39:32Z)
• Script-Agnostic Language Identification [21.19710835737713]
  Many modern languages, such as Konkani, Kashmiri, Punjabi etc., are synchronically written in several scripts. We propose learning script-agnostic representations using several different experimental strategies. We find that word-level script randomization and exposure to a language written in multiple scripts is extremely valuable for downstream script-agnostic language identification.
  arXiv  Detail & Related papers  (2024-06-25T19:23:42Z)
• The First Swahili Language Scene Text Detection and Recognition Dataset [55.83178123785643]
  There is a significant gap in low-resource languages, especially the Swahili Language. Swahili is widely spoken in East African countries but is still an under-explored language in scene text recognition. We propose a comprehensive dataset of Swahili scene text images and evaluate the dataset on different scene text detection and recognition models.
  arXiv  Detail & Related papers  (2024-05-19T03:55:02Z)
• Script Normalization for Unconventional Writing of Under-Resourced Languages in Bilingual Communities [36.578851892373365]
  Social media has provided linguistically under-represented communities with an extraordinary opportunity to create content in their native languages. This paper addresses the problem of script normalization for several such languages that are mainly written in a Perso-Arabic script. Using synthetic data with various levels of noise and a transformer-based model, we demonstrate that the problem can be effectively remediated.
  arXiv  Detail & Related papers  (2023-05-25T18:18:42Z)
• LIMIT: Language Identification, Misidentification, and Translation using Hierarchical Models in 350+ Languages [27.675441924635294]
  Current systems cannot accurately identify most of the world's 7000 languages. We first compile a corpus, MCS-350, of 50K multilingual and parallel children's stories in 350+ languages. We propose a novel misprediction-resolution hierarchical model, LIMIt, for language identification.
  arXiv  Detail & Related papers  (2023-05-23T17:15:43Z)
• READIN: A Chinese Multi-Task Benchmark with Realistic and Diverse Input Noises [87.70001456418504]
  We construct READIN: a Chinese multi-task benchmark with REalistic And Diverse Input Noises. READIN contains four diverse tasks and requests annotators to re-enter the original test data with two commonly used Chinese input methods: Pinyin input and speech input. We experiment with a series of strong pretrained language models as well as robust training methods, we find that these models often suffer significant performance drops on READIN.
  arXiv  Detail & Related papers  (2023-02-14T20:14:39Z)
• Speech-to-Speech Translation For A Real-world Unwritten Language [62.414304258701804]
  We study speech-to-speech translation (S2ST) that translates speech from one language into another language. We present an end-to-end solution from training data collection, modeling choices to benchmark dataset release.
  arXiv  Detail & Related papers  (2022-11-11T20:21:38Z)
• ASR2K: Speech Recognition for Around 2000 Languages without Audio [100.41158814934802]
  We present a speech recognition pipeline that does not require any audio for the target language. Our pipeline consists of three components: acoustic, pronunciation, and language models. We build speech recognition for 1909 languages by combining it with Crubadan: a large endangered languages n-gram database.
  arXiv  Detail & Related papers  (2022-09-06T22:48:29Z)
• Challenge Dataset of Cognates and False Friend Pairs from Indian Languages [54.6340870873525]
  Cognates are present in multiple variants of the same text across different languages. In this paper, we describe the creation of two cognate datasets for twelve Indian languages.
  arXiv  Detail & Related papers  (2021-12-17T14:23:43Z)
• Cross-lingual and Multilingual Spoken Term Detection for Low-Resource Indian Languages [13.42517182688574]
  Spoken Term Detection is the task of searching for words or phrases within audio, given either text or spoken input as a query. We use state-of-the-art Hindi, Tamil and Telugu ASR systems cross-lingually for lexical Spoken Term Detection in ten low-resource Indian languages. We show that it is possible to perform STD cross-lingually in a zero-shot manner without the need for any language-specific speech data.
  arXiv  Detail & Related papers  (2020-11-12T06:41:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.