MULTEXT-East

• URL: http://arxiv.org/abs/2003.14026v1
• Date: Tue, 31 Mar 2020 08:45:52 GMT
• Title: MULTEXT-East
• Abstract summary: This dataset is extensively documented, and freely available for research purposes. It covers 16 languages: Bulgarian, Croatian, Czech, English, Estonian, Hungarian, Macedonian, Persian, Polish, Resian, Romanian, Russian, Serbian, Slovak, Slovene, and Ukrainian. The parallel corpus, the novel "1984" by George Orwell, is sentence aligned and contains hand-validated morphosyntactic descriptions and lemmas.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: MULTEXT-East language resources, a multilingual dataset for language engineering research, focused on the morphosyntactic level of linguistic description. The MULTEXT-East dataset includes the EAGLES-based morphosyntactic specifications, morphosyntactic lexicons, and an annotated multilingual corpora. The parallel corpus, the novel "1984" by George Orwell, is sentence aligned and contains hand-validated morphosyntactic descriptions and lemmas. The resources are uniformly encoded in XML, using the Text Encoding Initiative Guidelines, TEI P5, and cover 16 languages: Bulgarian, Croatian, Czech, English, Estonian, Hungarian, Macedonian, Persian, Polish, Resian, Romanian, Russian, Serbian, Slovak, Slovene, and Ukrainian. This dataset is extensively documented, and freely available for research purposes. This case study gives a history of the development of the MULTEXT-East resources, presents their encoding and components, discusses related work and gives some conclusions.

Related papers

• VerChol -- Grammar-First Tokenization for Agglutinative Languages [0.0]
  Tokenization is the foundational step in all large language model (LLM) pipelines.<n>For agglutinative languages, a single word may encode root, tense, aspect, person, number, gender agreement, case, and postpositions into one orthographic unit.<n>Statistic tokenizers fragment these words into byte pair chunks that sever morpheme boundaries and token counts.
  arXiv  Detail & Related papers  (2026-03-06T04:07:15Z)
• Modeling Romanized Hindi and Bengali: Dataset Creation and Multilingual LLM Integration [0.3650590856721958]
  We introduce a novel transliteration dataset for two popular Indo-Aryan languages, Hindi and Bengali.<n>Our dataset comprises nearly 1.8 million Hindi and 1 million Bengali transliteration pairs.<n> Experimental results demonstrate significant improvements compared to existing relevant models in terms of BLEU and CER metrics.
  arXiv  Detail & Related papers  (2025-11-27T21:39:48Z)
• COMI-LINGUA: Expert Annotated Large-Scale Dataset for Multitask NLP in Hindi-English Code-Mixing [1.3062731746155414]
  COMI-LINGUA is the largest manually annotated dataset for code-mixed text, comprising 100,970 instances evaluated by three expert annotators in both Devanagari and Roman scripts. The dataset supports five fundamental NLP tasks: Language Identification, Matrix Language Identification, Part-of-Speech Tagging, Named Entity Recognition, and Translation. We evaluate LLMs on these tasks using COMILINGUA, revealing limitations in current multilingual modeling strategies and emphasizing the need for improved code-mixed text processing capabilities.
  arXiv  Detail & Related papers  (2025-03-27T16:36:39Z)
• Dialectal and Low-Resource Machine Translation for Aromanian [44.99833362998488]
  This paper presents the process of building a neural machine translation system with support for English, Romanian, and Aromanian. The primary contribution is the creation of the most extensive Aromanian-Romanian parallel corpus to date, consisting of 79,000 sentence pairs. To accomplish this, we introduce a suite of auxiliary tools, including a language-agnostic sentence embedding model for text mining and automated evaluation.
  arXiv  Detail & Related papers  (2024-10-23T10:00:23Z)
• MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling [70.34758460372629]
  We introduce a new paradigm that encodes the same information with segments of consistent size across diverse languages. MYTE produces shorter encodings for all 99 analyzed languages. This, in turn, improves multilingual LM performance and diminishes the perplexity gap throughout diverse languages.
  arXiv  Detail & Related papers  (2024-03-15T21:21:11Z)
• Cross-lingual Editing in Multilingual Language Models [1.3062731746155414]
  This paper introduces the cross-lingual model editing (textbfXME) paradigm, wherein a fact is edited in one language, and the subsequent update propagation is observed across other languages. The results reveal notable performance limitations of state-of-the-art METs under the XME setting, mainly when the languages involved belong to two distinct script families.
  arXiv  Detail & Related papers  (2024-01-19T06:54:39Z)
• $\mu$PLAN: Summarizing using a Content Plan as Cross-Lingual Bridge [72.64847925450368]
  Cross-lingual summarization consists of generating a summary in one language given an input document in a different language. This work presents $mu$PLAN, an approach to cross-lingual summarization that uses an intermediate planning step as a cross-lingual bridge.
  arXiv  Detail & Related papers  (2023-05-23T16:25:21Z)
• Romanization-based Large-scale Adaptation of Multilingual Language Models [124.57923286144515]
  Large multilingual pretrained language models (mPLMs) have become the de facto state of the art for cross-lingual transfer in NLP. We study and compare a plethora of data- and parameter-efficient strategies for adapting the mPLMs to romanized and non-romanized corpora of 14 diverse low-resource languages. Our results reveal that UROMAN-based transliteration can offer strong performance for many languages, with particular gains achieved in the most challenging setups.
  arXiv  Detail & Related papers  (2023-04-18T09:58:34Z)
• EUR-Lex-Sum: A Multi- and Cross-lingual Dataset for Long-form Summarization in the Legal Domain [2.4815579733050157]
  We propose a novel dataset, called EUR-Lex-Sum, based on manually curated document summaries of legal acts from the European Union law platform (EUR-Lex) Documents and their respective summaries exist as cross-lingual paragraph-aligned data in several of the 24 official European languages. We obtain up to 1,500 document/summary pairs per language, including a subset of 375 cross-lingually aligned legal acts with texts available in all 24 languages.
  arXiv  Detail & Related papers  (2022-10-24T17:58:59Z)
• XF2T: Cross-lingual Fact-to-Text Generation for Low-Resource Languages [11.581072296148031]
  We conduct an extensive study using popular Transformer-based text generation models on our extended multi-lingual dataset. Our experiments show that a multi-lingual mT5 model which uses fact-aware embeddings with structure-aware input encoding leads to best results on average across the twelve languages.
  arXiv  Detail & Related papers  (2022-09-22T18:01:27Z)
• Models and Datasets for Cross-Lingual Summarisation [78.56238251185214]
  We present a cross-lingual summarisation corpus with long documents in a source language associated with multi-sentence summaries in a target language. The corpus covers twelve language pairs and directions for four European languages, namely Czech, English, French and German. We derive cross-lingual document-summary instances from Wikipedia by combining lead paragraphs and articles' bodies from language aligned Wikipedia titles.
  arXiv  Detail & Related papers  (2022-02-19T11:55:40Z)
• Examining Cross-lingual Contextual Embeddings with Orthogonal Structural Probes [0.2538209532048867]
  A novel Orthogonal Structural Probe (Limisiewicz and Marevcek, 2021) allows us to answer this question for specific linguistic features. We evaluate syntactic (UD) and lexical (WordNet) structural information encoded inmBERT's contextual representations for nine diverse languages. We successfully apply our findings to zero-shot and few-shot cross-lingual parsing.
  arXiv  Detail & Related papers  (2021-09-10T15:03:11Z)
• A Multi-Perspective Architecture for Semantic Code Search [58.73778219645548]
  We propose a novel multi-perspective cross-lingual neural framework for code--text matching. Our experiments on the CoNaLa dataset show that our proposed model yields better performance than previous approaches.
  arXiv  Detail & Related papers  (2020-05-06T04:46:11Z)
• Multi-SimLex: A Large-Scale Evaluation of Multilingual and Cross-Lingual Lexical Semantic Similarity [67.36239720463657]
  Multi-SimLex is a large-scale lexical resource and evaluation benchmark covering datasets for 12 diverse languages. Each language dataset is annotated for the lexical relation of semantic similarity and contains 1,888 semantically aligned concept pairs. Owing to the alignment of concepts across languages, we provide a suite of 66 cross-lingual semantic similarity datasets.
  arXiv  Detail & Related papers  (2020-03-10T17:17:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.