Aksharantar: Open Indic-language Transliteration datasets and models for
the Next Billion Users
- URL: http://arxiv.org/abs/2205.03018v2
- Date: Thu, 26 Oct 2023 05:21:20 GMT
- Title: Aksharantar: Open Indic-language Transliteration datasets and models for
the Next Billion Users
- Authors: Yash Madhani, Sushane Parthan, Priyanka Bedekar, Gokul NC, Ruchi
Khapra, Anoop Kunchukuttan, Pratyush Kumar, Mitesh M. Khapra
- Abstract summary: We introduce Aksharantar, the largest publicly available transliteration dataset for Indian languages created by mining from monolingual and parallel corpora.
The dataset contains 26 million transliteration pairs for 21 Indic languages from 3 language families using 12 scripts.
Aksharantar is 21 times larger than existing datasets and is the first publicly available dataset for 7 languages and 1 language family.
- Score: 32.23606056944172
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Transliteration is very important in the Indian language context due to the
usage of multiple scripts and the widespread use of romanized inputs. However,
few training and evaluation sets are publicly available. We introduce
Aksharantar, the largest publicly available transliteration dataset for Indian
languages created by mining from monolingual and parallel corpora, as well as
collecting data from human annotators. The dataset contains 26 million
transliteration pairs for 21 Indic languages from 3 language families using 12
scripts. Aksharantar is 21 times larger than existing datasets and is the first
publicly available dataset for 7 languages and 1 language family. We also
introduce the Aksharantar testset comprising 103k word pairs spanning 19
languages that enables a fine-grained analysis of transliteration models on
native origin words, foreign words, frequent words, and rare words. Using the
training set, we trained IndicXlit, a multilingual transliteration model that
improves accuracy by 15% on the Dakshina test set, and establishes strong
baselines on the Aksharantar testset introduced in this work. The models,
mining scripts, transliteration guidelines, and datasets are available at
https://github.com/AI4Bharat/IndicXlit under open-source licenses. We hope the
availability of these large-scale, open resources will spur innovation for
Indic language transliteration and downstream applications. We hope the
availability of these large-scale, open resources will spur innovation for
Indic language transliteration and downstream applications.
Related papers
- Aya Dataset: An Open-Access Collection for Multilingual Instruction
Tuning [49.79783940841352]
Existing datasets are almost all in the English language.
We work with fluent speakers of languages from around the world to collect natural instances of instructions and completions.
We create the most extensive multilingual collection to date, comprising 513 million instances through templating and translating existing datasets across 114 languages.
arXiv Detail & Related papers (2024-02-09T18:51:49Z) - NusaWrites: Constructing High-Quality Corpora for Underrepresented and
Extremely Low-Resource Languages [54.808217147579036]
We conduct a case study on Indonesian local languages.
We compare the effectiveness of online scraping, human translation, and paragraph writing by native speakers in constructing datasets.
Our findings demonstrate that datasets generated through paragraph writing by native speakers exhibit superior quality in terms of lexical diversity and cultural content.
arXiv Detail & Related papers (2023-09-19T14:42:33Z) - Breaking Language Barriers: A Question Answering Dataset for Hindi and
Marathi [1.03590082373586]
This paper focuses on developing a Question Answering dataset for two such languages- Hindi and Marathi.
Despite Hindi being the 3rd most spoken language worldwide, and Marathi being the 11th most spoken language globally, both languages face limited resources for building efficient Question Answering systems.
We release the largest Question-Answering dataset available for these languages, with each dataset containing 28,000 samples.
arXiv Detail & Related papers (2023-08-19T00:39:21Z) - IndicTrans2: Towards High-Quality and Accessible Machine Translation
Models for all 22 Scheduled Indian Languages [37.758476568195256]
India has a rich linguistic landscape with languages from 4 major language families spoken by over a billion people.
22 of these languages are listed in the Constitution of India (referred to as scheduled languages)
arXiv Detail & Related papers (2023-05-25T17:57:43Z) - V\=arta: A Large-Scale Headline-Generation Dataset for Indic Languages [21.018996007110324]
This dataset includes 41.8 million news articles in 14 different Indic languages (and English)
To the best of our knowledge, this is the largest collection of curated articles for Indic languages currently available.
arXiv Detail & Related papers (2023-05-10T03:07:17Z) - ASR2K: Speech Recognition for Around 2000 Languages without Audio [100.41158814934802]
We present a speech recognition pipeline that does not require any audio for the target language.
Our pipeline consists of three components: acoustic, pronunciation, and language models.
We build speech recognition for 1909 languages by combining it with Crubadan: a large endangered languages n-gram database.
arXiv Detail & Related papers (2022-09-06T22:48:29Z) - Vakyansh: ASR Toolkit for Low Resource Indic languages [0.0]
Vakyansh is an end to end toolkit for Speech Recognition in Indic languages.
We create 14,000 hours of speech data in 23 Indic languages and train wav2vec 2.0 based pretrained models.
These pretrained models are then finetuned to create state of the art speech recognition models for 18 Indic languages.
arXiv Detail & Related papers (2022-03-30T17:50:18Z) - IndicNLG Suite: Multilingual Datasets for Diverse NLG Tasks in Indic
Languages [23.157951796614466]
In this paper, we present the IndicNLG suite, a collection of datasets for benchmarking Natural Language Generation for 11 Indic languages.
We focus on five diverse tasks, namely, biography generation using Wikipedia infoboxes (WikiBio), news headline generation, sentence summarization, question generation and paraphrase generation.
arXiv Detail & Related papers (2022-03-10T15:53:58Z) - XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning [68.57658225995966]
Cross-lingual Choice of Plausible Alternatives (XCOPA) is a typologically diverse multilingual dataset for causal commonsense reasoning in 11 languages.
We evaluate a range of state-of-the-art models on this novel dataset, revealing that the performance of current methods falls short compared to translation-based transfer.
arXiv Detail & Related papers (2020-05-01T12:22:33Z) - Multi-SimLex: A Large-Scale Evaluation of Multilingual and Cross-Lingual
Lexical Semantic Similarity [67.36239720463657]
Multi-SimLex is a large-scale lexical resource and evaluation benchmark covering datasets for 12 diverse languages.
Each language dataset is annotated for the lexical relation of semantic similarity and contains 1,888 semantically aligned concept pairs.
Owing to the alignment of concepts across languages, we provide a suite of 66 cross-lingual semantic similarity datasets.
arXiv Detail & Related papers (2020-03-10T17:17:01Z) - CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpus [57.641761472372814]
CoVoST is a multilingual speech-to-text translation corpus from 11 languages into English.
It diversified with over 11,000 speakers and over 60 accents.
CoVoST is released under CC0 license and free to use.
arXiv Detail & Related papers (2020-02-04T14:35:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.