Exploring Diachronic and Diatopic Changes in Dialect Continua: Tasks, Datasets and Challenges

• URL: http://arxiv.org/abs/2407.04010v1
• Date: Thu, 4 Jul 2024 15:38:38 GMT
• Title: Exploring Diachronic and Diatopic Changes in Dialect Continua: Tasks, Datasets and Challenges
• Authors: Melis Çelikkol, Lydia Körber, Wei Zhao,
• Abstract summary: We critically assess nine tasks and datasets across five dialects from three language families (Slavic, Romance, and Germanic) We outline five open challenges regarding changes in dialect use over time, the reliability of dialect datasets, the importance of speaker characteristics, limited coverage of dialects, and ethical considerations in data collection. We hope that our work sheds light on future research towards inclusive computational methods and datasets for language varieties and dialects.
• Score: 2.572144535177391
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Everlasting contact between language communities leads to constant changes in languages over time, and gives rise to language varieties and dialects. However, the communities speaking non-standard language are often overlooked by non-inclusive NLP technologies. Recently, there has been a surge of interest in studying diatopic and diachronic changes in dialect NLP, but there is currently no research exploring the intersection of both. Our work aims to fill this gap by systematically reviewing diachronic and diatopic papers from a unified perspective. In this work, we critically assess nine tasks and datasets across five dialects from three language families (Slavic, Romance, and Germanic) in both spoken and written modalities. The tasks covered are diverse, including corpus construction, dialect distance estimation, and dialect geolocation prediction, among others. Moreover, we outline five open challenges regarding changes in dialect use over time, the reliability of dialect datasets, the importance of speaker characteristics, limited coverage of dialects, and ethical considerations in data collection. We hope that our work sheds light on future research towards inclusive computational methods and datasets for language varieties and dialects.

Related papers

• Voices Unheard: NLP Resources and Models for Yorùbá Regional Dialects [72.18753241750964]
  Yorub'a is an African language with roughly 47 million speakers. Recent efforts to develop NLP technologies for African languages have focused on their standard dialects. We take steps towards bridging this gap by introducing a new high-quality parallel text and speech corpus.
  arXiv  Detail & Related papers  (2024-06-27T22:38:04Z)
• Disentangling Dialect from Social Bias via Multitask Learning to Improve Fairness [16.746758715820324]
  We present a multitask learning approach that models dialect language as an auxiliary task to incorporate syntactic and lexical variations. In our experiments with African-American English dialect, we provide empirical evidence that complementing common learning approaches with dialect modeling improves their fairness. Results suggest that multitask learning achieves state-of-the-art performance and helps to detect properties of biased language more reliably.
  arXiv  Detail & Related papers  (2024-06-14T12:39:39Z)
• What Do Dialect Speakers Want? A Survey of Attitudes Towards Language Technology for German Dialects [60.8361859783634]
  We survey speakers of dialects and regional languages related to German. We find that respondents are especially in favour of potential NLP tools that work with dialectal input.
  arXiv  Detail & Related papers  (2024-02-19T09:15:28Z)
• Natural Language Processing for Dialects of a Language: A Survey [56.93337350526933]
  State-of-the-art natural language processing (NLP) models are trained on massive training corpora, and report a superlative performance on evaluation datasets. This survey delves into an important attribute of these datasets: the dialect of a language. Motivated by the performance degradation of NLP models for dialectic datasets and its implications for the equity of language technologies, we survey past research in NLP for dialects in terms of datasets, and approaches.
  arXiv  Detail & Related papers  (2024-01-11T03:04:38Z)
• Task-Agnostic Low-Rank Adapters for Unseen English Dialects [52.88554155235167]
  Large Language Models (LLMs) are trained on corpora disproportionally weighted in favor of Standard American English. By disentangling dialect-specific and cross-dialectal information, HyperLoRA improves generalization to unseen dialects in a task-agnostic fashion.
  arXiv  Detail & Related papers  (2023-11-02T01:17:29Z)
• Quantifying the Dialect Gap and its Correlates Across Languages [69.18461982439031]
  This work will lay the foundation for furthering the field of dialectal NLP by laying out evident disparities and identifying possible pathways for addressing them through mindful data collection.
  arXiv  Detail & Related papers  (2023-10-23T17:42:01Z)
• End-to-End Automatic Speech Recognition model for the Sudanese Dialect [0.0]
  This paper comes to inspect the viability of designing an Automatic Speech Recognition model for the Sudanese dialect. The paper gives an overview of the Sudanese dialect and the tasks of collecting represented resources and pre-processing performed to construct a modest dataset. The designed model provided some insights into the current recognition task and reached an average Label Error Rate of 73.67%.
  arXiv  Detail & Related papers  (2022-12-21T07:35:33Z)
• Multi-VALUE: A Framework for Cross-Dialectal English NLP [49.55176102659081]
  Multi- Dialect is a controllable rule-based translation system spanning 50 English dialects. Stress tests reveal significant performance disparities for leading models on non-standard dialects. We partner with native speakers of Chicano and Indian English to release new gold-standard variants of the popular CoQA task.
  arXiv  Detail & Related papers  (2022-12-15T18:17:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.